Extracting Embedded ASN.1

Whilst strict ASN.1 [1] decoders exist, often the larger problem is that the data to be decoded is not delivered in a ‘pure’ ASN.1 format. Often additional records, such as headers and footers, are wrapped around the ASN.1 content making the file impossible to process with straight ASN.1 decoders.

For example, an Ericsson AXEGSM file’s records might include ‘FD – file descriptor’ (marking a file’s start), ‘LD – logical descriptor’ (marking CDR records from the same network element), ASN.1 formatted call details, ‘LT – logical terminator’ (making the end of an LD block of calls), and ‘FT – file trailer’ (marking the end of a file and its related control information).

FD File Header Record

LD Header Record

ASN.1 record (binary format #1)

ASN.1 record (binary format #2)

LT End Record

LD Header Record

ASN.1 record (binary format #2)

ASN.1 record (binary format #3)

LT End Record

FT File Trailer Record

Software that might process a regular non-ASN.1 file cannot tackle the file’s ASN.1 data, and ASN.1 decoders can’t address the additional (header/trailer) records spread within the file since they are not formatted in an ASN.1 layout. As an additional complication, these files may be compressed using zip and gzip to save disk storage and transmission times.

Tools such as CR-X can natively read such mixed-format files, identifying and separating out non-ASN.1 records and making use of any business data the records contain such as sequence numbers, data sources and file identifiers. The embedded ASN.1 records can be assessed and selectively passed to the ASN.1 decoder [2] for transformation into a contemporary format (e.g. CSV or fixed field-width records). Some ASN.1 records included in the source file may not be relevant for the processing context and can be omitted from processing before incurring a decoding effort penalty.

Once processed into a contemporary format, powerful data manipulation engines such as CR-X can be used to implement business rules against the ASN.1 and related data, generating records appropriate for immediate use within processing, or subsequent transmission to a downstream application for additional business use.

Please contact us for more information on how CR-X can be used to implement business rules against the ASN.1.

Additional Resources

[1] http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One

[2] CR-X Blog on ‘ASN.1 Integration

Categories: Product, Technical Tags: , , , , , , 0 Comments

JDBC/ODBC vs Native RDBMS Loaders

When faced with loading large quantities of data into a database, there are three main technologies available which are discussed here:

  • Vendor-supplied data loader
  • ODBC-based tools
  • JDBC-based tools



The choice will depend upon:

  • The number of database rows to be accessed (100′s, 1000′s, millions, etc.)
  • The type of access (row inserts, and/or updates/deletes.)
  • The frequency (daily, hourly, near real-time, etc.)
  • The database type, hardware and operating system



Most solutions will require a mixture of tools to provide a daily/weekly bulk load plus a stream of near real time updates during the day.

VENDOR SUPPLIED DATA LOADER

Positives

  • usually very fast, sometimes multi-threaded
  • usually included free with database licence



Negatives

  • data must be transformed to match input file format
  • usually does NOT support row update/delete
  • requires database software to be installed
  • requires vendor-supported hardware and operating system



ODBC-BASED TOOLS

Positives

  • supports row insert/update/delete
  • supports wide range of input file formats



Negatives

  • supported on a limited number of platforms only
  • supports a limited number of database types
  • requires vendor-supplied ODBC libraries
  • requires vendor-supported hardware and operating system



JDBC-BASED TOOLS (THIN CLIENT)

Positives

  • vendor-supplied JDBC driver is normally free
  • supports insert/update/delete row
  • supports wide range of input file formats
  • supports a wide range of database types
  • supports multi-threading
  • supported on a wide range of platforms (Java)



Negatives

  • requires vendor-supplied JDBC driver



BULK DATA LOAD

For the bulk data load, the vendor-supplied data loader will usually be fastest, though some tools such as CR-X (JDBC-based) can approach or even exceed native loader speed by using multi-threading and multiple connections to the database. Also, an ETL (extract, transform, load) tool will be needed to filter and transform the incoming transactions to produce a file format required by the vendor-supplied loader. It is best to avoid large footprint ETL tools that “stage” the data in a series of interim databases. Some small footprint ETL tools (such as CR-X) can do the filtering, transformation and data enrichment in a single step, and in-memory, dispensing with the need for staging databases.

Some vendor-supplied data loaders run much faster when simultaneously loading multiple files, so a file splitting tool such as CR-X can be used to evenly distribute transactions across multiple files and can allow the loading to proceed immediately (in parallel to the splitting).

NEAR REAL TIME UPDATES

For the near real time transactions during the day, a JDBC/ODBC-based tool will provide the addition requirement of updating/removing rows. Again, some ETL tools such as CR-X can do the filtering, transformation and data enrichment quickly and in-memory to provide near real time speed. Also keep in mind that incoming transactions could be a series of incoming files, or a single continuous feed (via TCP, UDP or Unix pipe) so you’ll need a tool such as CR-X which can handle these.

Categories: Product, Technical Tags: , , , , , 0 Comments

Small Footprint Technology for Business Object Histories

Complementary to the concept of a business object is the concept of a business object history. Classical system implementations of business objects persist their current state within the object. A history of a business object records:

  • the current state
  • the history of changes of state of the object
  • the events in which the object has participated, whether or not they result in a persisted change of state of the object itself.



Business object histories are a fundamental management tool, providing the factual basis for performance metrics, identification of behaviour patterns, and feedback on the impact of management actions.

The complexity in the representation of object histories arises in the event architecture within which the objects’ activities are portrayed. An event architecture comprises a general component applicable to all object types and a component specific to the types of business objects whose histories are to be recorded. We can describe each of these components as follows.

Three completely general event characteristics/relationships are desirable in any event architecture:

  • representation of a part/whole relationship between two events
  • representation of a sequence relationship between two events
  • representation of start time-date and end time-date for events, and thereby by implication a simple earlier/later time relationship.



The part/whole relationship should be recursive, or capable of nesting to an indefinite depth. The sequence relationship implies that one event is the predecessor and the other the successor as understood in the concept of a synchronous thread of execution. The sequence relationship is many-many, asymmetric, irreflexive, and implies a transitive earlier/later relation: it creates a partial ordering on the events it relates. Therefore the sequence relationship is suitable for representing the relationship between contributions [1]  in a business process, where after certain contributions parallel activity streams are created which eventually will come back together as predecessors of a later or final contribution in the process.

To complete a given event architecture, it is necessary to add the specific part/whole hierarchy of event types which will be represented. For instance a customer history might include all the business processes in which a given customer has participated with an organization. Recognized parts of a business process might include telephone calls which the customer has made to the organization’s call centre. Recognized parts of the telephone call might include segments, or those periods during the call when the organization as B-party is represented by a given respondent or group of respondents. Thus a given call might have as segment parts an initial period when the customer interacts with an IVR, followed by a segment when the customer is provided messages and music-on-hold by an ACD while awaiting the availability of a suitable consultant, followed by a segment talking with a targeted consultant, followed by a segment talking with both the consultant and a conferenced-in subject matter expert, and concluding, after a transfer, with a segment talking with a second consultant. It might further be desirable to recognize as parts of segments events such as authentication, promises to pay, the scheduling of an appointment, and so on.

Where an event type represents the interaction of two or more business object types, the event type will be re-useable within the histories for each of the business object types. For instance, taking our call-centre example from the previous paragraph, a segment of a telephone call might appear not only in the customer’s history, but also in the history or histories of each of the consultants participating in that segment.

When an event appears in the history of a given business object type, insights are available into the relations between that event and others in the history. For instance, when a telephone call appears in a customer history, there may or may not be earlier call events. Furthermore, any earlier calls may or may not be part of the same business process instance and may or may not be part of the same step or contribution of the given business process instance. It is only by having all these aspects of context represented that a truly meaningful metric of “repeat call” can be formulated. When a relationship such as being a repeat call has been detected, it is good practice to record the existence of the relationship within the representation of both the related events. By doing this, when the event representation is re-used in another business object history, where the context is no longer available, we do not lose sight of the existence of the relationship. For instance, when repeat calls have been identified within customer histories and appropriately recorded, and call segment representations are then re-used in consultant histories, the relevant information becomes available within the consultant histories to evaluate poor performance in failing to resolve customer issues and superior performance in resolving customer issues which previous consultants have failed to address.

In implementing a business object history, it is inevitable that information will need to be drawn from multiple source systems. Often even chains of contiguous events will traverse multiple operational systems. For instance representing a call in to a call-centre and all its parts may use information from CTI systems, IVR systems, and CRM systems. At the level of contributions, the events of a business process are typically non-contiguous and are managed in different systems addressing the different operational roles of the various participants. The lesson is that event representations are best rendered in a highly generic format, such as one composed of XML markup and content elements. Prefixed to such representations should be the content applicable to all events, that is the specification of part/whole and sequence relationships, identifiers, and event start and end times. The use of data from multiple systems poses further challenges of managing the association of events accessed in the different systems via local identifiers or indices, and managing differing latencies of data availability.

Because object histories quickly become very large, the choice of underlying technologies for their system representation requires care. CR-X has found it possible to develop file-based structures managed by the CR-X Transformation Engine which maintain consistency between the related histories of several interacting business object types. These structures significantly out-perform general purpose relational databases and enable histories several orders of magnitude larger than those achieved with relational technology.

In addition to the straightforward reporting and analytical purposes served by constructing business object histories, exciting new classes of systems become possible. For example ongoing work at CR-X uses object histories in the development and testing of predictive models of the behaviour of complex business objects, such as customers, corporations, or partners.

Footnote

[1] A contribution is the content of a single box in a workflow model of a business process. At a given level of decomposition of a business process it describes an outcome achieved by an individual, a group, or a system without interruption due to external dependencies.

Categories: Opinion, Product Tags: , , , , 0 Comments

Why has CRM failed?

Customer Relationship Management systems (CRMs) were launched on the market in the 1990s promising to provide the perfect 360° view of the customer.  In simple terms these new database systems were offering the ability to collect every single interaction with our clients and therefore lead to improved customer service and customer satisfaction.

15 years later very few CRM systems have lived up to their promise. There are many reasons why CRM systems have failed, but some reasons include:

  • Inability to match expectation with installation results
  • The re-use of CRM as merely a database of record (DBOR) for name, address and contact details
  • Replacement of centralized pricelists



The expectation of a centralized database CRM that would hold all customer interactions was always going to be a big ask.

However there is a way forward to reach these expectations.  It has now become a necessity.  All companies require a view of the end-to-end customer interactions and customer experiences.

To achieve these insights requires neither extensive development nor a new centralized data warehouse nor massive data migration.  It merely requires the addition of data integration capabilities that can extract customer interactions from existing operational systems near real-time, and then stitch these interactions into a single contiguous record.  This new concatenated customer interaction record outlines, in sequence, the interactions your clients have had with your business. When you measure these interaction records and match them against service level agreements and key performance indicators they will immediately alert you as to whether you have met your customer’s expectations.

Additionally when the customer interaction records are aggregated into “like” activities, they can be rolled-up to establish where there are major flaws in business processes and policies.  Black holes in your company’s procedures and systems are quickly highlighted, can be investigated and remedied.

CR-X has established a number of blueprints for end-to-end customer satisfaction collection and reporting systems.

Please contact us for more information on how CR-X can provide customer experience monitoring for your business.

Categories: Opinion Tags: , , , , 0 Comments

XML Integration

With the growing prevalence of XML (extensible markup language) data, there is a widening gulf between legacy systems and XML-based systems. There is a need for tools to bridge the gap between the two disparate groups, to allow the benefit of introducing newer XML-based systems whilst not incurring the cost of replacing large, long-running, and often reliable, legacy systems.

XML FORMAT

Data formatted using XML is somewhat readable by humans, and is very flexible. For example, a simple list of customers could be represented in XML as:

<?xml version=”1.0″ encoding=”UTF-8″ ?>
<Customers>
  <Customer custno=”10245″>
    <Name>John Smith</Name>
    <Phone>03 9111-2345</Phone>
  </Customer>
  <Customer custno=”10246″>
    <Name>Mary Jones</Name>
    <Phone>02 9123-4512</Phone>
  </Customer>
</Customers>

LEGACY FORMATS

However, data from legacy computer systems is wide-ranging in nature, and data encoding types include the following:

- ASCII (8-bit bytes)
- EBCDIC (8-bit bytes)
- UTF-8 (1, 2, 3 or 4-byte characters)
- BCD (4-bit nibbles)
- binary (8-bit, 16-bit, 32-bit, 64-bit)
- ASN.1 (mix of 8-bit bytes and 4-bit nibbles)
- odd/even bytes/nibbles swapped
- all bytes/nibbles/bits reversed
- values enclosed in quotes

Furthermore, legacy data field formats include the following:

- fixed width
- variable width, with field delimiter or length specifier
- space padded, zero padded, unpadded
- other padding characters (e.g. “F” nibble or “*” byte)
- padding on left/right
- numbers with left/right hand negative/positive signs
- floating point numbers with explicit/implicit decimal point
- date values in a wide range of formats and time zones

Far from using the same format for all fields in the file, legacy files often use a different format and/or encoding for each field!

Even the overall record can be formatted in various ways:

- fixed width
- specified length
- record delimiter
- more than one record type in the same file
- header record(s), including CSV header
- trailer record(s)

Enter the need for transformation tools! While there are many XML tools available, what is needed is a very flexible transformation tool such as CR-X which can be configured to interpret all of the above legacy formats
and can produce valid XML output.

Some legacy files can be very large (Terabytes!), and XML tools that load all the data into memory as a DOM (document object model) are usually limited to the amount of system memory and swap disk size. To overcome this problem, use a small footprint data streaming tool such as CR-X which reads the legacy data, and writes the XML data, one record at a time to avoid memory limitations.

Categories: Technical Tags: , , 0 Comments

Customer Experience Monitoring

“Our business is about technology, yes. But it’s also about operations and customer relationships” - Michael Dell -

Customer experience is the sum of all experiences a customer has with a supplier of goods or services, over the duration of their relationship with that supplier, from awareness, discovery, attraction, interaction, purchase, use, cultivation and advocacy.

Customer experience (positive, neutral or negative) can also simply be generated from an individual experience over one transaction.

Ensuring that your company generates positive customer experiences is of utmost importance! A survey in 16 countries, conducted by Alcatel/Lucent subsidiary Genesys, titled ‘The cost of poor customer service‘ highlighted that poor customer service, which therefore results in a bad customer experience, costs financial service companies, cable and satellite TV providers and telecommunications companies over $338.5 billion annually in lost business. All of these industries spend billions of dollars measuring internal systems, but not actual customer experience. If only one part of the internal system fails, the customer sees overall failure. Traditional business intelligence (BI) systems will not highlight these failures and therefore fail to measure the customer’s view of the service provided.

In order to really measure customer experience thus correcting BI system flaws you must assemble and end-to-end view. The results of doing so will encourage high quality customer self-service, reduced customer churn, lower operating costs, improved ARPU (average revenue per user) and long term customer loyalty.

Products like CR-X offer the ability to collect all of the individual interaction events such as;

  • Initial Contact via Call Centre, IVR or WebProduct, Service or Repair order processing and confirmation
  • Product or Service Provisioning
  • Activation and successful first billing

and ‘stitch’ them together in an Operational Data Store (ODS). This process will highlight individual events (triggers – exceptions), aggregate activity that breaches acceptable thresholds and allows you to monitor total customer experience. This in turn will provide the opportunity for continuous process and behavioural improvement and ultimately provide you with the raw material to gain a real insight into your customer experiences be that positive, neutral or negative.

For more information on how CR-X can provide customer experience monitoring for your business please contact us.

Categories: Opinion Tags: , 0 Comments

ASN.1 Integration

ASN.1 [1] (Abstract Syntax Notation One) is a flexible notation for representing data structures in telecommunications and computer networks. ASN.1 was originally defined in 1984, and has been updated over the intervening years with the latest revision released in 2002.

ASN.1 describes records in a ‘compact’ binary format that supports transmission of complex telecom and network data in a minimum of bytes whilst supporting highly optional field population. Pre-agreed ‘record’ definitions include information about field encoding (e.g. string, numeric and date formats), and allows scope for lists, field groups, and sub-lists/sub-groups to be specified.

ASN.1 records definitions which are similar to XML [3] that support nested definition structures, but without the tag names that make XML (human/machine) readable. In contrast ASN.1′s fields are labeled numerically (i.e. 1, 2, 3), and rely on details of what fields mean (e.g. ‘customer name’ versus ‘product code’) from previously agreed ‘record’ definitions.

There are a number of ways of encoding ASN.1 records, for example BER format [2]. In the BER format, each data field is described in terms of a type identifier, (value) length and value (TLV). The type identifier briefly and compactly identifies a field – in circumstances with highly optional field population, many fields may be unpopulated and not present in a specific data record. In one or more bytes, the length component identifies the length of the ‘value’ associated with the field.

Since the ASN.1 data is encoded in binary form, each component must be carefully read to address the possible ambiguities. For example, does a hexadecimal value in a specific byte (i.e. x’04′) mean the start of field four (T), a field value ‘length’ (L) of four bytes, or a binary field value (V) of four?

Whilst ASN.1 encoded data is primarily used within telecommunication and computer networks, it can also be passed from them to business applications that perform for example assurance reporting and monitoring. These data interfaces when passed from the network for downstream use may be wrapped with header/trailer records and compressed (e.g. gzip) to save disk space and transmission time.

The issue with ASN.1 is that many data tools are unable to read its binary formats natively. Raw ASN.1 data must be transformed from its binary encoding into a readable format such as CSV or fixed field-width records. A solution to this issue is CR-X, which is able to read and manipulate regular mid-range and legacy data formats, can decode ASN.1 binary files into an accessible form for subsequent processing. Business rules can be applied to the decoded files before they are passed to downstream business applications for subsequent use.

Where multiple record formats exist in the ASN.1 source files, records can be omitted when not required, and compressed files are addressed natively by CR-X processing.

Additional Resources

[1] http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One

[2] http://en.wikipedia.org/wiki/Basic_Encoding_Rules

[3] http://en.wikipedia.org/wiki/XML

Categories: Technical Tags: , , , , , , 1 Comment

Processing Genesys Files

T-Server records are generated by the network equipment that underpins Call Centre Routing, capturing technical details of agent and CTI-related handoffs. T-Server records are logged verbosely to high-volume files, with new files being initiated every few minutes by each of the many CTI applications, at each of the numerous instances that support a call centre. Once a T-Server file’s initial log processing is complete, the log files are often sent to an internal archive location/server where they are often compressed to save storage space, allowing many days to be stored online.

The information logged in T-Server files is voluminous and captures many attributes of the infrastructure equipment, routing, call centre applications, agent handoffs and customer details – only specific fields of the many available are of interest to downstream business applications that might make use of T-Server sourced data.

The archived T-Server data must be retrieved from its ‘network’ location to a server where sufficient processing capability can be provided, before the extracted results are handed off to the business applications that use it. The retrieval of these files can be performed as a large daily effort, or as a series of smaller retrievals performed incrementally during the day using tools such as rsync.

Reliable processing of the high data-volumes involved and picking the fields required are key to making business use of retrieved T-Server log files. Source T-Server records must be ‘boiled down’ to include just specific fields (which may change over time), and apply the required business rules of the downstream applications.

CR-X’s high performance and versatile data configurations can be generated quickly to support T-Server log processing. Demanding requirements of downstream business applications can be adapted from CR-X’s existing template configurations.

Business applications that might use T-Server extracts include:

- Call Centre Analytics (analysing the paths customers take when interacting with a call centre), and

- Network Monitoring (preemptively looking for problems in the infrastructure that underpins a business’s call centre operations).

Different business applications can have varied processing properties. For example:

- Call Centre Analytics looks at all calls across a range of days to identify trends and usability problems in call centre processing, and

- Network Monitoring applications focus on what is happening ‘now’ rather than what happened earlier today or yesterday.

In addition to generating extracts for downstream business applications, additional ‘value add’ processing can be performed across T-Server and related data sources/files. For example, connection identifiers, uniquely generated for each inbound call to a call centre, can be associated with additional values generated by different equipment, or when handset transfers occur between agents. Without such an association, the component parts of an inbound call will be analysed separately instead of being viewed as a whole.

For more information on how CR-X’s high performance and versatile data configurations can be utilized to support your T-Server log processing please contact CR-X.

Additional Resource

[1] http://wiki.answers.com/Q/WHAT_IS_a_genesys_t-server

Categories: Technical Tags: , , , , , 0 Comments