Casing and text search/comparison tutorial

I have recently received a question from one of my blog readers.
He was asking about the possibility of doing case-sensitive search in AX.

I would like to reply to that by creating a small tutorial on search in AX. It is definitely not going to cover all the scenarios, but will give beginners a basic understanding of their options.

Download the tutorial xpo from my Skydrive

The tutorial consists of 1 form with 3 tab pages.

1.
Comparing two strings in AX is very simple: You can basically use the equality operator "==". As you can see from the tutorial, this is the case-insensitive operations, so "vanya" is equal to "Vanya".
AX also supports case-sensitive comparison. Kernel function strCmp() compares two strings taking into account the casing of the symbols in the string.

2.
Searching for a substring is a common operation in AX.
For that you have a number of options, as usual:
strScan() function ignores casing and allows you to specify from which position to search, and how many symbols. This is very basic, and a method like this is present in any programming language.
TextBuffer.find() is a more advanced use of the search mechanism. First of all, it allows to ignore or take into account the casing in the source text. Similar to strScan, it allows to specify the start position for the search.
What it also has is support for Regular Expressions, as well as the ability to, for example, paste the text to Windows clipboard.
Lastly, there is the match() function. The main purpose of it is to find a match based on the specified pattern using regular expressions, but nothing prevents using it for a simple search operation. It has a rather limited output though. You only get a boolean value stating whether a match was found or not, while with the previous 2 methods you also get the position of the substring.

Interesting
An interesting discovery that I made when writing the tutorial was about the speed of the different search operations. I have included this into the tutorial, so you can go in and try it yourself on your specific setup.
On my box, strScan() was the slowest operation of all, while TextBuffer, which I considered to be a very heavy class, was performing rather well.

Of course, single operation time compared to database operations is very low, so you won't notice it in your daily work. But it is something to think about.

3.
Finding a symbol in a string based on a specified set of symbols is also possible.
You have 2 functions at your disposal for that: strFind and strNFind. The difference is that strNFind searches for any symbol NOT present in the provided set, compared to strFind.


This is not an extensive list, so I would be interested in hearing which functions you use or what your results for performance comparison would be.

UtcDateTime in Dynamics AX 2009

In Dynamics AX 2009, Microsoft introduced a new data type, UtcDateTime, that is going to eventually replace the 2 existing types, Date and Time, which are still present in the application right now.
Obviously, the introduction of this new type requires a tutorial on how it can be used on forms, how you can filter on fields of this type, as well as what functions are available out of the box for it.
So I have made such a tutorial, and I hope it will be useful for developers upgrading to AX 2009.

Download the xpo for the tutorial from my SkyDrive

The tutorial consists of a single form, containing the following elements:

• a grid, displaying data from CustTable
• 4 buttons for various filtering actions
• 3 controls that allow specifying the filtering conditions for the data

. In the form, you can see how UtcDateTime based controls are displayed both in a regular group and in a grid.



Below is an explanation of the implemented functionality, in form of a Question/Answer section:

1. Q: Can I filter on the new UtcDateTime type, specifying the Date part only?
   A: Yes. You simply have to specify only the date part when applying the filter, like below. Note, that this also works fine when filtering directly from the UI (Ctrl+F).
   

   qbdsCustTable.addRange(fieldNum(CustTable, CreatedDateTime)).value(queryValue(DateFilter.dateValue()));

   What is interesting is how the kernel processes this range. In the below infolog, you can see that when viewing the query, it displays a "==" condition on a specific dateTime value.
   But in reality, as you can see from the SQL trace, a range ">= && <=" condition is applied to span exactly one day.
   Also note, that the values in the trace are displayed accounting for the TimeZone I am in, as well as for Daylight Saving Time
   
   
2. Q: Can I filter on the new UtcDateTime type, specifying the Time part only?
   A: No, this is not possible with a UtcDateTime type. The range applied when specifying a Time value is the minimum DateTime value, as seen below. Note, that in the SQL trace it is converted to "no range".
   
   
3. Q: Can I use similar query functions for UtcDateTime type?
   A: Yes. All the main existing functions for working with QueryBuildRange also support UtcDateTime. For example, in the infolog below you can see how a range on 2 UtcDateTime dates is applied. Global::queryRange method was used to achieve that. Note, again, that the SQL trace offsets the DateTime by the appropriate number of hours based on my location.
   
   
4. Q: How is the UtcDateTime stored in the database? Is it displayed the same way on forms?
   A: The UtcDateTime fields are in the database always stored in Coordinated Universal time (UTC). Whenever displayed on forms and bound to table fields, the data is converted to the user's preferred timezone. Note, that you need to take care of the conversion yourself, if the control is not bound to a field. For an example, see the init method of the tutorial form.
5. Q: What standard helper functions are present for working with UtcDateTime type in the application?
   A: The main entry point for working with UtcDateTime type is the DateTimeUtil class. It allows adding Days/Months/Years, as well as applying an offset, getting the user's preffered timezone, converting from/to other types, etc. An example from the form init method is posted below:
   

       // getSystemDateTime() returns the current DateTime set in the system, not the current machine dateTime.
       // Note that getSystemDateTime() returns a UTC date and time, not your local date time.
       // In order to receive your local DateTime value, you should use methods applyTimeZoneOffset and specify the preferred time zone.
       utcDateTimeFilter.dateTimeValue(
           DateTimeUtil::applyTimeZoneOffset(
               DateTimeUtil::getSystemDateTime(),
               DateTimeUtil::getUserPreferredTimeZone()));

6. Q: Does this mean that the support for Date and Time types has been removed?
   A: No, Date and Time are still supported. As you can see in the form init method, SystemDateGet(), timeNow(), today() are all still supported
7. Q: I don't see the actual filter values in the SQL log. Instead, all I see are "?"'s. Also, how can I limit the number of data/fields selected from the database?
   A: This is just some extra stuff, not related to UtcDateTime, but still useful to know and pay attention to.
   CustTable has a very large number of fields, and I am only displaying 4 of those in the form, so it would be unwise to always query and return all of the fields. Luckily, the datasource has a property OnlyFetchActive, which controls the query behavior by only selecting the fields actually displayed on the form. Note, that you should avoid using this with editable datasources. See comments to this post for details
   As for "?"'s in the SQL trace - that is happening due to the use of placeholders. This in general optimizes the performance of the queries, by creating a query execution plan and storing it for future use. But it is possible, and is required in some specific cases, to force the use of literals (meaning the actual values of the ranges in the query). This can be done using the literals method on the query. See method init on the form for an example.

Performance optimization: Deleting inventory journal lines

Preamble

As a developer, you should always consider performance implications of the code you write. In an ERP application like Microsoft Dynamics AX, the main focus should be on query execution, since it takes up the overwhelming part of the servers' resources.

You should always write queries that would execute the minimum amount of time and use the minimum amount of resources, at the same time producing the expected output in all cases.

Note, that performance is one of those things you cannot really verify on a 1-box install with a small test dataset. Most query problems show up only when tested with many users concurrently loading the AOS on a large-size database.

Code example

Table method
AOT\Data Dictionary\Tables\InventJournalTrans\Methods\delete
contains the following code:

    if (this.Voucher)
    {
        if (this.numOfVoucherLines() == 0)
            JournalError::deleteVoucher(tablenum(InventJournalTable),this.JournalId,this.Voucher);
    }

The code is logically correct, deleting related records from the JournalError table, which contains error messages generated during validation and posting of journals.
But now let us consider the actual implementation. If we rephrase the conditions under which we delete the error message history, it would sound something like:

If a Voucher number is specified on the line being deleted, and there is no more lines in this journal that use this Voucher number, then we remove the JournalError records.

Deeper code analysis for performance

So, let's go in deeper, and open the code for method numOfVoucherLines:

Integer numOfVoucherLines()
{
    return any2int((select count(RecId) from inventJournalTrans
                      where inventJournalTrans.JournalId == this.JournalId &&
                            inventJournalTrans.Voucher   == this.Voucher).RecId);
}

As you can see, it counts all the journal lines with the specified Voucher number and JournalId.
Does it generate the expected output? Yes.
Is it optimal for the scenario in which it is used? No.

In order to determine if JournalError records can be deleted, we only need to know if at least 1 record with the specified Voucher number exists. So why do we need to search and count all such records? We don't.
Note that this code is being executed for each journal line, magnifying the impact of a non-optimal query N-fold.

Suggested solution

So, if we create a new method like this:

boolean voucherLineExist()
{
    return (select firstfast firstonly forceplaceholders recId from inventJournalTrans
            index hint VoucherIdx
            where inventJournalTrans.journalId == this.journalId &&
                  inventJournalTrans.voucher   == this.voucher).recId != 0;
}

and use it instead of numOfVoucherLines() in delete() method, we will greatly improve the overall performance when deleting journal lines.
It has been measured, that the time it takes to delete a journal with a large number of lines (88000, to be specific) has reduced by 6 times, which is fantastic.

Notes

• forceplaceholders is used in the query explicitly to make sure a query plan is created and reused for this query, which further improves the performance, since the Voucher number most probably changes from line to line (depends on related journal name setup).
• It is generally still a best practice to avoid creating inventory journals with such a large number of rows, because the journal is usually processed in 1 database transaction.

See also

• Link to AxForum, where this problem was originally reported and solution suggested.
• Link to Microsoft Dynamics AX Performance team blog, containing useful information about performance tuning and tools