Development tutorial: Extensibility: Replaceable in Chain of Command methods

Recently we announced a new and pretty powerful Extensibility feature, wrapping methods with Chain of Command in augmentation classes. This allows to write much cleaner extensions with fewer lines of code, as well as provides some extra capabilities like access to protected fields and methods of augmented object, easier way of ensuring a single transaction scope for standard and extension code, etc.

If you are not yet familiar with this feature, you are missing out. Go read about it:
https://docs.microsoft.com/en-us/dynamics365/unified-operations/dev-itpro/extensibility/method-wrapping-coc

There was one significant restriction applied (by design) to these wrapper methods:

Wrapper methods must always call next

Wrapper methods in an extension class must always call next, so that the next method in the chain and, finally, the original implementation are always called. This restriction helps guarantee that every method in the chain contributes to the result.

However, what this resulted in is a more complex implementation and "workaround-like" solutions in standard code to enable some of the commonly requested extension points, where the ISV/VAR would like to completely replace the standard logic with an alternative implementation that does the same or a very similar operation.


With Platform update 11 we have added a new attribute, which allows Microsoft (on request from multiple partnres), where it is justified, to decorate a particular protected or public method, allowing wrapper methods to not call next on it, replacing the logic of that method.

Here's how it looks:

/// 

/// Attribute used to enable or disable replacing a method in an extension class. 
/// 
/// 
/// Private methods can not be set to be replaceable even with the usage of this attribute.
/// 
public class ReplaceableAttribute extends SysAttribute
{
    boolean isReplaceable;

    public void new(boolean _isReplaceable = true)
    {
        super();
        this.isReplaceable = _isReplaceable;
    }
}

Example

OK, let's now look at an example of how this will be used.

Note. Since the attribute only appeared in PU11, that means that all application released up to and including Spring release 2017 do not have any methods marked with this attribute. It is only now with the Fall release of 2017 that you might see some methods being tagged this way.


Say, an ISV wanted to provide an alternative implementation for looking up Warehouses on a specified Site, more specifically, for the method InventLocation.lookupBySiteIdAllTypes().
One way to solve this could be to add a delegate, invoke it at the beginning of the method, and then check the EventHandlerAcceptResult to see if someone has replaced the implementation, in which case, short-circuit the method execution, so standard logic is not executed.

A potential implementation shown below:

public class InventLocation extends common
{
    public static void lookupBySiteIdAllTypes(FormStringControl _ctrl, InventSiteId _inventSiteId)
    {
        EventHandlerAcceptResult lookupBySiteIdResult = EventHandlerAcceptResult::newSingleResponse();
        InventLocation::lookupBySiteIdAllTypesDelegate(_ctrl, _inventSiteId, lookupBySiteIdResult);

        if (lookupBySiteIdResult.isAccepted())
        {
            return;
        }

        SysTableLookup sysTableLookup = SysTableLookup::newParameters(tableNum(InventLocation), _ctrl);
        ListEnumerator listEnumerator = List::create(InventLocation::standardLookupFields()).getEnumerator();

        while (listEnumerator.moveNext())
        {
            sysTableLookup.addLookupfield(fieldName2id(tableNum(InventLocation), listEnumerator.current()));
        }

        sysTableLookup.parmQuery(InventLocation::standardLookupBySiteIdQuery(_inventSiteId));
        sysTableLookup.performFormLookup();
    }
}

Lookups is one of the common examples, where people might was a complete replacement of the standard logic. Note that by definition that means only one of the ISV solutions can replace it. If two attempt to accept() the result, an error will be shown.
That would typically mean that a logical conflict exists between the two ISV solutions, and the VAR would need to decide which ones to use, or make it configurable somehow.

Now, let's try to see what could be done with the new attribute, if Microsoft were to apply it on this method.

public class InventLocation extends common
{
    [Replaceable]
    public static void lookupBySiteIdAllTypes(FormStringControl _ctrl, InventSiteId _inventSiteId)
    {
        SysTableLookup sysTableLookup = SysTableLookup::newParameters(tableNum(InventLocation), _ctrl);
        ListEnumerator listEnumerator = List::create(InventLocation::standardLookupFields()).getEnumerator();

        while (listEnumerator.moveNext())
        {
            sysTableLookup.addLookupfield(fieldName2id(tableNum(InventLocation), listEnumerator.current()));
        }

        sysTableLookup.parmQuery(InventLocation::standardLookupBySiteIdQuery(_inventSiteId));
        sysTableLookup.performFormLookup();
    }
}

The ISV can now in his model wrap this method in an augmentation class, provide his own implementation, and avoid calling next():

Important. 
We recommend to always make the call conditional, so that your own logic that is not calling next is only invoked for your specific case. This will make you a good citizen, that can co-exist with other ISV solutions also wrapping the same method.

[ExtensionOf(tableStr(InventLocation))]
public final class MyPU11InventLocationTable_Extension
{
    public static void lookupBySiteIdAllTypes(FormStringControl _ctrl, InventSiteId _inventSiteId)
    {
        const str MySpecialWarehouseCtrlName = 'MySpecialWarehouseCtrl';
        if (!_inventSiteId || _ctrl.name() == MySpecialWarehouseCtrlName)
        {
            // Your own logic
            _ctrl.performTypeLookup(extendedTypeNum(InventLocationId));
        }
        else
        {
            next lookupBySiteIdAllTypes(_ctrl, _inventSiteId);
        }
    }
}

Pretty easy and neat, huh?

Missing an extension point? Log it!

Again, remember, that in order to skip calling next, the method needs to be marked by Microsoft as Replaceable.
If you need a particular method to be Replaceable, or if you in general need an extension point that is not available in the latest available release, please follow the instructions outlined here to create an extensibility request for us.

Links

To review the list of features included in Platform update 11, see the What's new or changed topic and refer to the KB article for details about the customer found bug fixes included in this update.

Development Tutorial: Extensibility: Adding a table display/edit method and showing it on a form in PU11

In my previous post I described the capabilities of the Dynamics 365 FOE platform update 10, when it comes to working with display/edit methods.

In Platform Update 11 a few improvements came out, which I will describe below (with an example).

Let's use the same example as in my previous post, and add a new display method showing the internal product name for a selected product - we'll compose it by appending some text to the product search name.

Step 1 - Create a table extension and add a new display method to it - New recommended approach

All the approaches described in the previous post are still applicable, but are, in my opinion, not as intuitive as the below, so I'd recommend to always use the below approach.

[ExtensionOf(tableStr(EcoResProduct))]
public final class MyPU11_EcoResProductExtensionOfTable_Extension
{
    [SysClientCacheDataMethod]
    public display Name myInternalProductName()
    {
        return "PU11: " + this.SearchName;
    }

    // This is same as in PU10
    [SysClientCacheDataMethod]
    public static display Name myInternalProductNameStatic(EcoResProduct _ecoResProduct)
    {
        return "PU11 static: " + _ecoResProduct.SearchName;
    }
}
K/code>

OK, so what do we have here?
A simple instance method, no redundant arguments, access to table fields and methods through this.
Just as with overlayering, there's really no difference.

Note. The logic of the method is not really important - you'd have your fields used here, most probably, but for the sake of the example I just use SearchName field.

Step 2 - Add the method to a form through a table field group

So now let's see another thing, which was not possible before PU11.

Let's create an extension for the metadata changes of the EcoResProduct table, and add a new Field Group. After that we'll put our new display method in it, as shown below.

Select the new display method as the source for the new field group field.

As you can see from the image above, you can now use the drop-down, and it will show you all the display/edit methods created in Extension/Augmentation classes as well as the standard ones.

The syntax is the same as described last time:

• :: for static methods
• . for instance methods (new)

All that's left is to add the new field group onto a form.

Step 3 - Add the methods to a form through extension

Same as before, we create a form extension, and add the new fields to it.

With field groups it is as easy as with overlayering - you can just drag and drop the new field group from the DataSources\EcoResProduct node onto the Design of the form, and it will create the new group and sub-controls, and set the appropriate properties on them, as shown below:

Add new field group onto the form extension

As easy as that.

You can also add the fields bound to the new display methods directly, as shown before. Except now you can also use the instance display/edit methods, which wasn't available earlier.

Add an instance extension display method to a form

Limitations

• The drop-down for display methods does not show the extension methods, as with field groups. This should be addressed in one of the upcoming updates.
• You are still not able to declare the display method on the form itself, or on the form data source.

Conclusion

Some last words: The future is bright! :) 

The tooling is getting better with every release, and thanks to the monthly updates and binary compatibility of the releases, new innovation from Microsoft is just a month away!

Links

You can download the project from the example from my OneDrive.

To review the list of features included in Platform update 11, see the What's new or changed topic and refer to the KB article for details about the customer found bug fixes included in this update.

Development Tutorial: Extensibility: Adding a table display/edit method and showing it on a form in PU10

One of the super common tasks for an application developer working to address customer requirements is adding display methods showing some additional customer-specific information on existing forms.

Usually you would overlayer the corresponding table and form and insert the missing method. Overlayering is not an option soon, however it is possible to do the same using only Extensions.

As an example, let us add a new display method showing the internal product name for a selected product - we'll compose it by appending some text to the product search name.

Step 1 - Create a table extension and add a new display method to it - Option 1

As you know, there are two ways to create extension classes now, so let's see both ways in action.
Here we'll look at the "old" way, where we create an actual extension class (as in .NET), so it must be static, and the display method must be static as well, and take the record as the first argument.

Here's how it looks for my example:

/// 

/// Extension class for EcoResProduct table.
/// 
public static class MyPU10_EcoResProductTable_Extension
{
    [SysClientCacheDataMethod]
    public static display Name myInternalProductName(EcoResProduct _ecoResProduct)
    {
        return 'IntName: ' + strReplace(_ecoResProduct.SearchName, ' ', '');
    }
}

As you can see, we can define the above method and it will compile even though normally declaring a static display method is not allowed by the compiler.
We can also decorate the method with attribute, like I have done here by applying the display method caching attribute.
The logic of the method is not really important - you'd have your fields used here, most probably, but for the sake of the example I just use SearchName field.

Step 2 - Create a table extension and add a new display method to it - Option 2

So, another "new" way to extend a table is through an augmentation class, using the ExtensionOf attribute. This is shown below for my example:

/// 

/// Extension class for EcoResProduct table.
/// 
[ExtensionOf(tableStr(EcoResProduct))]
final class MyPU10_EcoResProductExtensionOfTable_Extension
{
    public static display Name myInternalProductName(EcoResProduct _ecoResProduct)
    {
        return "Alt: " + _ecoResProduct.SearchName;
    }
}

As you can see, this again is a static method - declaring it as an instance method will compile and would allow you to reference the record through this, but you will not be able to use it as a display method on the form as of today.
The method also needs to take the record as the argument.
It, of course, can also be decorated with the SysClientCacheDataMethod attribute, as in the first example.

Step 3 - Add the methods to a form through extension

Note - Limitation

You cannot as of today add the newly created display methods to a field group on the table. It will compile, but the control will not show up on the form if you add the field group to it.

First off, we'll need to create an extension of the EcoResProductDetails form in the desired model.

Then we'll add a new tab page to it and place two new String controls into it.

Now, the trick is with how to specify the display method name in Properties.

See the example below:

Specify properties for form string control to bind it to a table data method

As you can see, the trick is to specify the full name, including the class name and the method name with the static method delimiter in the format:

<class name>::<static method name>

This is only supported for table methods, so you won't be able to do the same for a Form Data Source, for example.

Result

Here's how our new awesome display methods look at run-time:

Additional information shown through display methods on Product details form

Download the project

You can download it from my OneDrive here.

What's next

In an upcoming platform update we hope to provide a much more intuitive way of adding display methods, however the above approach will keep being supported.

Stay tuned for an update!

Development tutorial: insert_recordset using the Query class

Introduction

I am sure most of you are familiar with set-based X++ CUD operators: insert_recordset, update_recordset and delete_from. They allow performing database operations with a large number of records in a single roundtrip to the server, instead of a row-by-row type of operation, which depends on the number of rows being processed. As a result, they can provide a very significant boost in overall performance of a selected flow.

If not familiar or just need a refresher, you can read more about it by following the link to MSDN.

Problem statement

There's however one significant drawback with these operators - they are all compile-time constructs, so they lack the flexibility of the flow modifying the database request before it is set based on runtime state of the flow.

And, once the application is sealed, that will mean there is no way to modify the request even at compile-time, as X++ statements are not extensible.

Solution

In Microsoft Dynamics AX 2012 R3 a static method was added on the Query class, that allows to solve the two problems above for insert_recordset. This is of course now also available in Microsoft Dynamics 365 for Finance and Operations: Enterprise edition.

Note: update_recordset and delete_from are still not supported through the Query class.

Example

Imagine that we need to write a function that would copy selected sales order line information into a sales order line history table for one or more orders.

1. Data model

Here's how the data model for this table is defined in this example:

DEV_SalesLineHistory data model diagram

2. New DEV_SalesLineHistory table in Visual Studio

And here's how the new table looks in Visual Studio (I created a new model for it in a separate package dependent on Application Suite):

DEV_SalesLineHistory table

Note I skipped all stuff non-essential to this example

3. Copy function through regular X++ insert_recordset statement

Let's first write the statement for inserting the history records using the regular insert_recordset operator:

public class DEV_Tutorial_InsertRecordset
{
    public static Counter insertXppInsert_Recordset(SalesId _salesId)
    {
        DEV_SalesLineHistory    salesLineHistory;
        SalesLine               salesLine;
        InventDim               inventDim;

        JournalPostedDateTime   postedDateTime = DateTimeUtil::utcNow();
        JournalPostedUserId     postedBy = curUserId();
        SalesDocumentStatus     postingType = DocumentStatus::PackingSlip;

        insert_recordset salesLineHistory
        (
            SalesId,
            LineNum,
            InventTransId,
            SalesQty,
            SalesUnit,
            InventSiteId,
            InventLocationId,
            PostedDateTime,
            PostedBy,
            PostingType
        )
        select SalesId, LineNum, InventTransId, SalesQty, SalesUnit from salesLine
            where salesLine.SalesId == _salesId
            join InventSiteId, InventLocationId, postedDateTime, postedBy, postingType from inventDim
                where inventDim.InventDimId == salesLine.InventDimId;

        return any2int(salesLineHistory.RowCount());
    }
}

As you can see, we do a simple select from SalesLine, specifying the exact fields, joined to selected fields from InventDim, where the field list also contains a few local variables to populate into the rows being inserted.
This is the standard syntax used with X++ insert_recordset statement, which all of you are familiar with.

4. Method signature for Query::insert_recordset()

Now let's convert the above to a Query, and call Query::insert_recordset() instead.
This method accepts three arguments:

• An instance of a table record. This is where data will be inserted into. We can then use this variable to ask how many rows were inserted, for example.
• An instance of a Map(Types::String, Types::Container), which defines the mapping of the fields to copy. In X++ operator, this had to be based on the specific order in the field selection lists in the select statement. 
• The map key is the target field name.
• The value is a container, which defines a pair of values:
• the unique identifier of the QueryBuildDataSource object points to the table to copy the value from
• the field name on the above data source to copy the value from
• An instance of a Query class, which defines the select statement for the data, similar to what you see in the X++ version above.

As you can see from the above, it does not account for literals, as we did with variables in the X++ operator example.
That is currently not supported with this API.
We can however solve this through a use of a "temporary" table, as suggested below.

5. Define a new table to store calculable literals

Let us define a new table that will store the data required by our insert statement. That means it needs to contain four fields:

• PostedDateTime
• PostedBy
• PostingType
• SalesId - we'll use this to join to SalesLine. This could be sessionId or whatever is required to ensure concurrency and uniqueness

Here's how the table would look in Visual Studio designer:

DEV_SalesLineHistoryPostingDataTmp table definition

We can now populate this table with the required values and join it to our query.

After executing the bulk insert we can then delete the inserted row (if necessary).

Another possible implementation here could be to use a View, with computed  columns for the different literal values needed. You could select from a table that has only 1 row, like InventParameters or the like. This is however less flexible, as it'll be compiled in, while with a "temporary" table you could determine the values at runtime.

6. Write up the code using the Query::insert_recordset() method

Now we are all set to write the necessary code. It would look like below:

public class DEV_Tutorial_InsertRecordset
{
    public static Counter insertQueryInsert_Recordset(SalesId _salesId)
    {
        DEV_SalesLineHistory    salesLineHistory;
        
        Query query = new Query();
        QueryBuildDataSource qbdsSalesLine = query.addDataSource(tableNum(SalesLine));
        qbdsSalesLine.addSelectionField(fieldNum(SalesLine, SalesId));
        qbdsSalesLine.addSelectionField(fieldNum(SalesLine, LineNum));
        qbdsSalesLine.addSelectionField(fieldNum(SalesLine, InventTransId));
        qbdsSalesLine.addSelectionField(fieldNum(SalesLine, SalesQty));
        qbdsSalesLine.addSelectionField(fieldNum(SalesLine, SalesUnit));
        qbdsSalesLine.addRange(fieldNum(SalesLine, SalesId)).value(queryValue(_salesId));
        QueryBuildDataSource qbdsInventDim = qbdsSalesLine.addDataSource(tableNum(InventDim));
        qbdsInventDim.addSelectionField(fieldNum(InventDim, InventLocationId));
        qbdsInventDim.addSelectionField(fieldNum(InventDim, InventSiteId));
        qbdsInventDim.relations(true);
        QueryBuildDataSource qbdsPostingData = qbdsInventDim.addDataSource(tableNum(DEV_SalesLineHistoryPostingDataTmp));
        qbdsPostingData.addLink(fieldNum(SalesLine, SalesId), fieldNum(DEV_SalesLineHistoryPostingDataTmp, SalesId), qbdsSalesLine.name());
        qbdsPostingData.addSelectionField(fieldNum(DEV_SalesLineHistoryPostingDataTmp, PostedDateTime));
        qbdsPostingData.addSelectionField(fieldNum(DEV_SalesLineHistoryPostingDataTmp, PostedBy));
        qbdsPostingData.addSelectionField(fieldNum(DEV_SalesLineHistoryPostingDataTmp, PostingType));

        Map targetToSourceMap = new Map(Types::String, Types::Container);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, SalesId),           [qbdsSalesLine.uniqueId(), fieldStr(SalesLine, SalesId)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, LineNum),           [qbdsSalesLine.uniqueId(), fieldStr(SalesLine, LineNum)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, InventTransId),     [qbdsSalesLine.uniqueId(), fieldStr(SalesLine, InventTransId)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, SalesQty),          [qbdsSalesLine.uniqueId(), fieldStr(SalesLine, SalesQty)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, SalesUnit),         [qbdsSalesLine.uniqueId(), fieldStr(SalesLine, SalesUnit)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, InventLocationId),  [qbdsInventDim.uniqueId(), fieldStr(InventDim, InventLocationId)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, InventSiteId),      [qbdsInventDim.uniqueId(), fieldStr(InventDim, InventSiteId)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, PostedDateTime),    [qbdsPostingData.uniqueId(), fieldStr(DEV_SalesLineHistoryPostingDataTmp, PostedDateTime)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, PostedBy),          [qbdsPostingData.uniqueId(), fieldStr(DEV_SalesLineHistoryPostingDataTmp, PostedBy)]);
        targetToSourceMap.insert(fieldStr(DEV_SalesLineHistory, PostingType),       [qbdsPostingData.uniqueId(), fieldStr(DEV_SalesLineHistoryPostingDataTmp, PostingType)]);

        ttsbegin;
        
        DEV_SalesLineHistoryPostingDataTmp postingData;
        postingData.PostedDateTime = DateTimeUtil::utcNow();
        postingData.PostedBy = curUserId();
        postingData.PostingType = DocumentStatus::Invoice;
        postingData.SalesId = _salesId;
        postingData.insert();
        
        Query::insert_recordset(salesLineHistory, targetToSourceMap, query);

        delete_from postingData 
            where postingData.SalesId == _salesId;

        ttscommit;

        return any2int(salesLineHistory.RowCount());
    }
}

As you can see, the first part of the code builds a query using the QueryBuild* class hierarchy. The query is identical to the above select statement, with the addition of another join to our "tmp" table to retrieve the literals.
The second part populates the target to source Map object, which maps the fields to insert into to their source.
The third part actually invokes the operation, making sure we have the record populated in our "tmp" table beforehand.

Note. Because the first argument is the record we insert into, we can use it to get the RowCount(), telling us how many records have been inserted.

7. Extensibility aspect

Leaving the code just like it is does not actually make it extensible, as the partners would not be able to add additional fields to copy, or additional conditions/joins to the query. To accomplish this, we'd need to break the logic out into smaller methods with specific responsibilities, or add inline delegates to do the same. Generally speaking, you should always favor breaking the code down into simpler methods over delegates.

I've not done this in the example, as that's not the purpose, but you should follow these guidelines in production code.

8. Execute and test the code

We can execute these methods now, but first we need to ensure we have a test subject, aka a Sales order to practice with. Here's the one I used in the example:

A sales order with 3 order lines

And here's the Runnable Class that invokes the two methods above:

class DEV_Tutorial_InsertRecordset_Runner
{        
    public static void main(Args _args)
    {        
        const SalesId SalesId = '000810';

        if (SalesTable::exist(SalesId))
        {
            Counter xppInsert = DEV_Tutorial_InsertRecordset::insertXppInsert_Recordset(SalesId);
            Counter queryInsert = DEV_Tutorial_InsertRecordset::insertQueryInsert_Recordset(SalesId);

            strFmt('Tutorial Insert_Recordset');
            info(strFmt('Inserted using X++ insert_recordset = %1', xppInsert));
            info(strFmt('Inserted using Query::insert_recordset() = %1', queryInsert));
        }
    }

}

You can download the full project here.


Hope this helps! Let me know if you have any comments.

Announcement: Plan of record design for InventDim extensibility

As you all hopefully know by now, we are on a journey towards overlayering-free solutions.
One big roadblock on this path was the Inventory dimension concept we have in AX, or, more specifically, providing a way for partners to add new inventory dimensions that would not require overlayering.

Michael has documented the current design we have in mind in a blog post you can read below:
https://blogs.msdn.microsoft.com/mfp/2017/08/10/extensible-inventory-dimensions/

Let me know if you have any questions!

Announcement: Extensibility documentation is now available

With the next release of Dynamics 365 for Finance and Operations Enterprise edition we plan to soft seal the Application Suite model.
For anyone not yet familiar with this agenda, please read through the previous announcement

The partner channel need to be educated for them to successfully migrate their development to using extensions instead of overlayering.

In order to help with that we have created a dedicated section on Extensibility in our documentation, which has a number of details about our plans, the process of migrating to extensions, as well as specific How to articles around platform capabilities, as well as specific application frameworks and how to extend them (coming soon...)

Home page for Extensibility documentation

If you cannot find a particular topic, let us know, either directly on the docs page, or here through comments to this post.

Hope this helps!

Thanks

Development tutorial link: Extending the Warehouse management functionality after application seal

Background

At the Dynamics 365 for Operations Technical Conference earlier this year, Microsoft announced its plans around overlayering going forward. If you have not heard it yet, here's the tweet I posted on it:

 Follow

Ivan Kashperuk @IvanKashperuk

#Dyn365Tech AppSuite will be "soft sealed" in Fall 2017 release and "hard sealed" in Spring 2018 - move to use #Extensions in your solutions

2:25 AM - 14 Mar 2017

 
•  1313 Retweets
 
•  22 likes

However the application SYS code still has a number of areas which are difficult to impossible to customize without overlayering. One of such areas is the Warehouse management area.

Improvements

In the SCM team we have been focusing on improving our Extensibility story in the application, which also includes the above mentioned warehousing area.

Below are a few posts Michael posted recently describing some of the changes we have done in the Warehouse management area, which are now available as part of Microsoft Dynamics for Operations Spring 2017 preview.
This should allow customizing the warehousing functionality to a large degree without the need to overlayer the code.

Extending WHS – Adding a new flow
Extending WHS – Adding a new control type
Extending WHS – Changing behavior of existing control types
Extending WHS – Adding a new custom work type
Extending WHS – Adding a new location directive strategy

Please look through these changes and let us know if YOUR functional enhancements are still impossible, so we can get them addressed.

Thanks

Development tutorial: SysExtension framework with SysExtensionIAttribute and an Instantiation strategy

Problem statement

This post will continue from where we left off in my previous blog post about the SysExtension framework: Development tutorial: SysExtension framework in factory methods where the constructor requires one or more arguments

In the above blog post I described how to use the SysExtension framework in combination with an Instantiation strategy, which applies in many cases where the class being instantiated requires input arguments in the constructor.

At the end of the post, however, I mentioned there is one flaw with that implementation. That problem is performance.

If you remember a blog post by mfp from a while back (https://blogs.msdn.microsoft.com/mfp/2014/05/08/x-pedal-to-the-metal/), in it he describes the problems with the SysExtension framework in AX 2012 R2, where there were two main issues:

• A heavy use of reflection to build the cacheKey used to look up the types
• Interop impact when needing to make Native AOS calls instead of pure IL

The second problem is not really relevant in Dynamics 365 for Operations, as everything runs in IL now by default.

And the first problem was resolved through introduction of an interface, SysExtensionIAttribute, which would ensure the cache is built by the attribute itself and does not require reflection calls, which immediately improved the performance by more than 10x.

Well, if you were paying attention to the example in my previous blog post, you noticed that my attribute did not implement the above-mentioned interface. That is because using an instantiation strategy in combination with the SysExtensionIAttribute attribute was not supported.

It becomes obvious if you look at the comments in the below code snippet of the SysExtension framework:

public class SysExtensionAppClassFactory extends SysExtensionElementFactory
{
    ...
    public static Object getClassFromSysAttribute(
        ClassName       _baseClassName,
        SysAttribute    _sysAttribute,
        SysExtAppClassDefaultInstantiation  _defaultInstantiation = null
        )
    {
        SysExtensionISearchStrategy         searchStrategy;
        SysExtensionCacheValue              cachedResult;
        SysExtModelAttributeInstance        attributeInstance;
        Object                              classInstance;
        SysExtensionIAttribute              sysExtensionIAttribute = _sysAttribute as SysExtensionIAttribute;

        // The attribute implements SysExtensionIAttribute, and no instantiation strategy is specified
        // Use the much faster implementation in getClassFromSysExtAttribute().
        if (sysExtensionIAttribute && !_defaultInstantiation)
        {
            return SysExtensionAppClassFactory::getClassFromSysExtAttribute(_baseClassName, sysExtensionIAttribute);
        }
        ...
    }
    ...
}

So if we were to use an Instantiation strategy we would fall back to the "old" way that goes through reflection. Moreover, it would actually not work even then, as it would confuse the two ways of getting the cache key.

That left you with one of two options:

• Not implement the SysExtensionIAttribute on the attribute and rip the benefits of using an instantiation strategy, but suffer the significant performance hit it brings with it, or
• Use the SysExtensionIAttribute, but as a result not be able to use the instantiation strategy, which limited the places where it was applicable

No more! 

We have updated the SysExtension framework in Platform Update 5, so now you can rip the benefits of both worlds, using an instantiation strategy and implementing the SysExtensionIAttribute interface on the attribute.

Let us walk through the changes required to our project for that:

1.

First off, let's implement the interface on the attribute definition. We can now also get rid of the parm* method, which was only necessary when the "old" approach with reflection was used, as that was how the framework would retrieve the attribute value to build up the cache key. 

class NoYesUnchangedFactoryAttribute extends SysAttribute implements SysExtensionIAttribute
{
    NoYesUnchanged noYesUnchangedValue;

    public void new(NoYesUnchanged _noYesUnchangedValue)
    {
        noYesUnchangedValue = _noYesUnchangedValue;
    }

    public str parmCacheKey()
    {
        return classStr(NoYesUnchangedFactoryAttribute)+';'+int2str(enum2int(noYesUnchangedValue));
    }

    public boolean useSingleton()
    {
        return true;
    }

}

As part of implementing the interface we needed to provide the implementation of a parmCacheKey() method, which returns the cache key taking into account the attribute value. We also need to implement the useSingleton() method, which determines if the same instance should be returned by the extension framework for a given extension.

The framework will now rely on the parmCacheKey() method instead of needing to browse through the parm methods on the attribute class.

2.

Let's now also change the Instantiation strategy class we created, and implement the SysExtensionIInstantiationStrategy interface instead of extending from SysExtAppClassDefaultInstantiation. That is not necessary now and is cleaner this way.

public class InstantiationStrategyForClassWithArg implements SysExtensionIInstantiationStrategy
{
...
}

The implementation should stay the same.

3. 

Finally, let's change the construct() method on the base class to use the new API, by calling the getClassFromSysAttributeWithInstantiationStrategy() method instead of getClassFromSysAttribute() (which is still there for backward compatibility):

public class BaseClassWithArgInConstructor
{
...
    public static BaseClassWithArgInConstructor construct(NoYesUnchanged _factoryType, str _argument)
    {
        NoYesUnchangedFactoryAttribute attr = new NoYesUnchangedFactoryAttribute(_factoryType);
        BaseClassWithArgInConstructor inst = SysExtensionAppClassFactory::getClassFromSysAttributeWithInstantiationStrategy(
            classStr(BaseClassWithArgInConstructor), attr, InstantiationStrategyForClassWithArg::construct(_argument));
        
        return inst;
    }
}

Result

Running the test now will produce the following result in infolog:

The derived class was returned with the argument populated in

Download

You can download the full project for the updated example from my OneDrive.

Hope this helps!

Development tutorial link: Extensibility challenges: Pack/Unpack in RunBase classes

Introduction + example

As you know, we have been focusing on extending our Extensibility story in the application, as well as trying to document the various patterns common to the application and how to address them if you are an ISV and need to extend some existing functionality.

mfp has recently written a blog post describing how you can extend the information shown on a RunBase-based dialog, and how to handle that information once the user enters the necessary data.

You can read through that particular example here: 

https://blogs.msdn.microsoft.com/mfp/2017/01/31/extending-class-state/

What that example did not describe is how to preserve the user entered data, so that next time the dialog is opened, it contains the last entries already populated. This is the typical pattern used across all AX forms and is internally based on the SysLastValue table.

In RunBase classes it is done through the pack and unpack methods (as well as initParmDefault).
For ensuring seemless code upgrade of these classes they also rely on a "version" of the stroed SysLastValue data, which is typically stored in a macro definition. The RunBase internal class state that needs to be preserved between runs is typically done through a local macro.
A typical example is shown below (taken from the Tutorial_RunBaseBatch class):

    #define.CurrentVersion(1)
    #localmacro.CurrentList
        transDate,
        custAccount
    #endmacro

    public container pack()
    {
        return [#CurrentVersion, #CurrentList];
    }

    public boolean unpack(container packedClass)
    {
        Version version = RunBase::getVersion(packedClass);
    
        switch (version)
        {
            case #CurrentVersion:
                [version,#CurrentList] = packedClass;
                break;
            default:
                return false;
        }

        return true;
    }

Just in short, what happens is that:

• We save the packed state of the class with the corresponding version into the SysLastValue table record for this class, which means that all variables in the CurrentList macro need to be "serializable". 
• The container will look something like this: [1, 31/3/2017, "US-0001"]
• When we need to retrieve/unpack these values, we retrieve the version as we know it's the first position in the container.
• If the version is still the same as the current version, read the packed container into the variables specified in the local macro
• If the version is different from the current version, return false, which will subsequently run initParmDefault() method to load the default values for the class state variables 

Problem statement

This works fine in overlayering scenarios, because you just add any additional state to the CurrentList macro and they will be packed/unpacked when necessary automatically.

But what do you do when overlayering is not an option? You use augmentation / extensions.

However, it is not possible to extend macros, either global or locally defined. Macros are replaced with the corresponding text at compile time which would mean that all the existing code using the macros would need to be recompiled if you extended it, which is not an option.

OK, you might say, I can just add a post-method handler for the pack/unpack methods, and add my additional state there to the end of the container.

Well, that might work if your solution is the only one, but let's look at what could happen where there are 2 solutions side by side deployed:

1. Pack is run and returns a container looking like this (Using the example from above): [1, 31/3/2017, "US-0001"]
2. Post-method handler is called on ISV extension 1, and returns the above container + the specific state for ISV 1 solution (let's assume it's just an extra string variable): [1, 31/3/2017, "US-0001", "ISV1"]
3. Post-method handler is called on ISV extension 2, and returns the above container + the specific state for ISV 2 solution: [1, 31/3/2017, "US-0001", "ISV1", "ISV2"]

Now, when the class is run the next time around, unpack needs to be called, together with the unpack method extensions from ISV1 and ISV2 solutions.

1. Unpack is run and assigns the variables from the packed state (assuming it's the right version) to the base class variables.
2. ISV2 unpack post-method handler is called and needs to retrieve only the part of the container which is relevant to ISV2 solution
3. ISV1 unpack post-method handler is called and needs to do the same 

Steps 2 and 3 cannot be done in a reliable way. OK, say we copy over the macro definitions from the base class, assuming also the members are public and can be accessed from our augmentation class or we duplicate all those variables in unpack and hope nothing changes in the future :) - and in unpack we read the sub-part of the container from the base class into that, but how can we ensure the next part is for our extension? ISV1 and ISV2 post-method handlers are not necessarily going to be called in the same order for unpack as they were for pack.

All in all, this just does not work.

Note

The below line is perfectly fine in X++ and will not cause issues, which is why the base unpack() would not fail even if the packed container had state for some of the extensions as well.

[cn, value, value2] = ["SomeClass", 4, "SomeValue", "AnotherClass", true, "more values"];

The container being assigned has more values than the left side.

Solution

In order to solve this problem and make the behavior deterministic, we came up with a way to uniquely identify each specific extension packed state by name and allow ISVs to call set/get this state by name.

With Platform Update 5 we have now released this logic at the RunBase level. If you take a look at that class, you will notice a few new methods:

• packExtension - adds to the end of the packed state (from base or with other ISV extensions) container the part for this extension, prefixing it with the name of the extension
• unpackExtension - look through the packed state container and find the sub-part for this particular extension based on extension name.
• isCandidateExtension - evaluates if the passed in container is possible an extension packed state. For that it needs to consist of the name of the extension + packed state in a container.

You can read more about it and look at an example follow-up from mfp's post below:

https://ax.help.dynamics.com/en/wiki/extend-the-runbase-class/

Hope this helps!

Development tutorial: Platform improvements for handling X++ exceptions the right way

A while back I wrote about a pattern we discovered in X++ code which could lead to serious data consistency issues. You can read it again and look at an example mfp wrote up here:
http://kashperuk.blogspot.dk/2016/11/tutorial-link-handling-exceptions-right.html

With the release of Platform update 5 for Dynamics 365 for Operations we should now be better guarded against this kind of issue.

Let's look at the below example (needs to be run in USMF company):

class TryCatchAllException
{
    public static void main(Args _args)
    {
        setPrefix("try/catch example");

        try
        {
            ttsbegin;

            TryCatchAllException::doSomethingInTTS();

            ttscommit;
        }
        catch
        {
            info("Inside main catch block");
        }

        info(strfmt("Item name after main try/catch block: %1", InventTable::find("A0001").NameAlias));
    }

    private static void doSomethingInTTS()
    {
        try
        {
            info("Doing something");
   
            InventTable item = InventTable::find("A0001", true);
            item.NameAlias = "Another name";
            item.doUpdate();

            throw Exception::UpdateConflict;

            // Some additional code was supposed to be executed here
        }
        catch
        {
            info("Inside doSomething catch block");
        }
  
        info("After doSomething try/catch block");

    }

}

Before Platform Update 5 the result would be:

As you can see, we 

• went into doSomething(), 
• executed the update of NameAlias for the item, 
• then an exception of type UpdateConflict was thrown
• At this point the catch-all block caught this exception without aborting the transaction, meaning the item is still updated. We did not abort the transaction because we did not think about this case before
• We exit the doSomething() and commit the transaction, even though we got an exception and did not want anything committed (because the second part of the code did not execute)
• As a result, the NameAlias is still modified.

Now with Platform Update 5 the result will be:

That is, we

• went into doSomething(),
• executed the update of NameAlias for the item,
• then an exception of type UpdateConflict was thrown
• At this point the catch-all did not catch this type of exception, as we are inside a transaction scope, so the exception was unhandled in this scope, and went to the one above
• Since we are by the outer scope outside the transaction, the catch-all block caught the exception,
• and the NameAlias is unchanged

So, again, we will simply not handle the 2 special exception types (UpdateConflict and DuplicateKey) any longer in a catch-all block inside a transaction scope, you will either need to handle them explicitly or leave it up to the calling context to handle.

This will ensure we do not get into this erroneous code execution path where the transaction is not aborted, but we never handle the special exception types internally.

Hope this helps!

Development tutorial: SysExtension framework in factory methods where the constructor requires one or more arguments

Background

At the Dynamics 365 for Operations Technical Conference earlier this week, Microsoft announced its plans around overlayering going forward. If you have not heard it yet, here's the tweet I posted on it:

#Dyn365Tech AppSuite will be "soft sealed" in Fall 2017 release and "hard sealed" in Spring 2018 - move to use #Extensions in your solutions

— Ivan Kashperuk (@IvanKashperuk) March 14, 2017

The direct impact of this change is that we should stop using certain patterns when writing new X++ code.

Pattern to avoid

One of these patterns is the implementation of factory methods through a switch block, where depending on an enumeration value (another typical example is table ID) the corresponding sub-class is returned.

First off, it's coupling the base class too tightly with the sub-classes, which it should not be aware of at all.
Secondly, because the application model where the base class is declared might be sealed (e.g, foundation models are already sealed), you would not be able to add additional cases to the switch block, basically locking the application up for any extension scenarios.

So, that's all good and fine, but what can and should we do instead?

Pattern to use

The SysExtension framework is one of the ways of solving this erroneous factory method pattern implementation. 

This has been described in a number of posts already, so I won't repeat here. Please read the below posts instead, if you are unfamiliar with how SysExtension can be used:

• https://blogs.msdn.microsoft.com/mfp/2013/06/12/sysextension-framework-to-the-rescue/
• https://blogs.msdn.microsoft.com/ax_gfm_framework_team_blog/2012/11/13/the-microsoft-dynamics-ax-2012-extension-framework-part-1/

Problem statement

In many cases in existing X++ code, the constructor of the class (the new() method) takes one or more arguments. In this case you cannot simply use the SysExtension framework methods described above. 

Here's an artificial example I created to demonstrate this:

• A base class that takes one string argument in the constructor. This could also be abstract in many cases.
• Two derived classes that need to be instantiated depending on the value of a NoYesUnchanged enum passed into the construct() method.

public class BaseClassWithArgInConstructor
{
    public str argument;

    
    public void new(str _argument)
    {
        argument = _argument;
    }

    public static BaseClassWithArgInConstructor construct(NoYesUnchanged _factoryType, str _argument)
    {
        // Typical implementation of a construct method
        switch (_factoryType)
        {
            case NoYesUnchanged::No:
                return new DerivedClassWithArgInConstructor_No(_argument);
            case NoYesUnchanged::Yes:
                return new DerivedClassWithArgInConstructor_Yes(_argument);
        }

        return new BaseClassWithArgInConstructor(_argument);
    }
}

public class DerivedClassWithArgInConstructor_No extends BaseClassWithArgInConstructor
{
}

public class DerivedClassWithArgInConstructor_Yes extends BaseClassWithArgInConstructor
{
}

And here's a Runnable class we will use to test our factory method:

class TestInstantiateClassWithArgInConstructor
{        
    public static void main(Args _args)
    {        
        BaseClassWithArgInConstructor instance = BaseClassWithArgInConstructor::construct(NoYesUnchanged::Yes, "someValue");
        setPrefix("Basic implementation with switch block");
        info(classId2Name(classIdGet(instance)));
        info(instance.argument);
    }
}

Running this now would produce the following result:

The right derived class with the correct argument value returned

OK, so to decouple the classes declared above, I created a "factory" attribute, which takes a NoYesUnchanged enum value as input.

public class NoYesUnchangedFactoryAttribute extends SysAttribute
{
    NoYesUnchanged noYesUnchangedValue;

    public void new(NoYesUnchanged _noYesUnchangedValue)
    {
        noYesUnchangedValue = _noYesUnchangedValue;
    }

    public NoYesUnchanged parmNoYesUnchangedValue()
    {
        return noYesUnchangedValue;
    }
}

Let's now decorate the two derived classes and modify the construct() on the base class to be based on the SysExtension framework instead of the switch block:

[NoYesUnchangedFactoryAttribute(NoYesUnchanged::No)]
public class DerivedClassWithArgInConstructor_No extends BaseClassWithArgInConstructor
{
}

[NoYesUnchangedFactoryAttribute(NoYesUnchanged::Yes)]
public class DerivedClassWithArgInConstructor_Yes extends BaseClassWithArgInConstructor
{
}

public class BaseClassWithArgInConstructor
{
    // ...
    public static BaseClassWithArgInConstructor construct(NoYesUnchanged _factoryType, str _argument)
    {
        NoYesUnchangedFactoryAttribute attr = new NoYesUnchangedFactoryAttribute(_factoryType);
        BaseClassWithArgInConstructor instance = SysExtensionAppClassFactory::getClassFromSysAttribute(classStr(BaseClassWithArgInConstructor), attr);

        return instance;
    }
}

Running the test now will however not produce the expected result:

The right derived class is returned but argument is missing

That is because by default the SysExtension framework will instantiate a new instance of the corresponding class (dictClass.makeObject()), which ignores the constructor arguments.

Solution

In order to account for the constructor arguments we need to use an Instantiation strategy, which can then be passed in as the 3rd argument when calling SysExtensionAppClassFactory.

Let's define that strategy class:

public class InstantiationStrategyForClassWithArg extends SysExtAppClassDefaultInstantiation
{
    str arg;

    public anytype instantiate(SysExtModelElement  _element)
    {
        SysExtModelElementApp   appElement = _element as SysExtModelElementApp;
        Object                  instance;

        if (appElement)
        {
            SysDictClass dictClass = SysDictClass::newName(appElement.parmAppName());
            if (dictClass)
            {
                instance = dictClass.makeObject(arg);
            }
        }

        return instance;
    }

    protected void new(str _arg)
    {
        this.arg = _arg;
    }

    public static InstantiationStrategyForClassWithArg construct(str _arg)
    {
        return new InstantiationStrategyForClassWithArg(_arg);
    }
}

As you can see above, we had to

• Define a class extending from SysExtAppClassDefaultInstantiation (it's unfortunate that it's not an interface instead). 
• Declare all of the arguments needed by the corresponding class we plan to construct.
• Override the instantiate() method, which is being invoked by the SysExtension framework when the times comes
• In there we create the new object instance of the appElement and, if necessary, pass in any additional arguments, in our case, arg.

Let's now use that in our construct() method:

public class BaseClassWithArgInConstructor
{
    //...
    public static BaseClassWithArgInConstructor construct(NoYesUnchanged _factoryType, str _argument)
    {
        NoYesUnchangedFactoryAttribute attr = new NoYesUnchangedFactoryAttribute(_factoryType);
        BaseClassWithArgInConstructor instance = SysExtensionAppClassFactory::getClassFromSysAttribute(
            classStr(BaseClassWithArgInConstructor), attr, InstantiationStrategyForClassWithArg::construct(_argument));

        return instance;
    }
}

If we now run the test, we will see the following:

The right derived class with the correct argument value returned 

Note

If you modify the attributes/hierarchy after the initial implementation, you might need to clear the cache, and restarting IIS is not enough, since the cache is also persisted to the DB. You can do that by invoking the below static method:

SysExtensionCache::clearAllScopes();

Parting note

There is a problem with the solution described above. The problem is performance.

I will walk you through it, as well as the solution, in my next post.

Announcement: Warehouse Mobile Devices Portal improvements with February update of AX 2012 R3

A monthly cadence AX 2012 R3 update for February has just come out on LCS, and with it a few changes our team has done that deal with overall behavior and performance of WMDP - the Warehouse Mobile Devices Portal used in the Advanced warehousing module.

I want to call them out here, and if you are reading my blog to keep up to date with the Warehousing changes, I strongly encourage you to install the below changes:

• A functional enhancement, that allows you to start execution of a work order that has some lines awaiting demand replenishment, processing the lines that can be picked already now
• Can also be downloaded individually as KB3205828
• Make sure to pair it up with KB3209909, if you use the Full button when executing work
• You can read more about the functionality in my earlier blog post: http://kashperuk.blogspot.dk/2016/11/tutorial-link-executing-outbound-work.html
• A performance optimization, that avoids updating certain persisted counters on the wave when work goes through its stages. 
• This also ensures they do not get out of sync, as they were used to make certain decisions about what is allowed for a work order
• Can also be downloaded individually as KB3217157
• An integrity enhancement, that ensures we always are in a valid state in DB, where each service call from WMDP is now executed within a single transaction scope
• Can also be downloaded individually as KB3210293
• Can be turned off in code for a selected WHSWorkExecuteDisplay* class, if needed
• This is a great change ensuring we do not commit any data unless all went well, but might hypothetically impact your new/modified WMDP flows, if you handle your exceptions incorrectly today. If you do find issues with them, please let me know, I am curious to know your specific examples.
• Various minor enhancements, that ensure WMDP performs well under load
• Can also be downloaded individually as KB3210293
• This includes improvements to enable better concurrency in various WMDP scenarios, better error handling on user entry, etc.

Again, install these, try them out, and provide feedback!

There are a lot more enhancements and bug fixes that went into this release, you can read the full list by following the link below:

https://blogs.technet.microsoft.com/dynamicsaxse/2017/03/01/february-release-dynamics-ax-2012-r3/

Thanks!

Extensible enums: Breaking change for .NET libraries that you need to be aware of

A while back I wrote a blog post describing the Extensible Enums - a new feature that is part of Dynamics 365 for Operations:
http://kashperuk.blogspot.com/2016/09/development-tutorial-extensible-base.html

I explained that when marking an enumeration as extensible, the representation of this enum under the hood (in CLR) changes. Here's the specific quote:

The extensible enums are represented in CLR as both an Enum and a Class, where each enum value is represented as a static readonlyfield. So accessing a specific value from the above enum, say, NumberSeqModule::Invent would under the hood look something like NumberSeqModule_Values.Invent, where Invent is of type NumberSeqModule which is an Enum. It would in turn call into AX to convert the specific named constant "Invent" to its integer enumeration value through a built-in function like the symbol2Value on DictEnum.

Something that was not super clear in the post is that this was actually a breaking change that might impact your .NET solutions relying on one of these base enumerations.

Problem statement

As part of enabling further extensibility in the application for the next release, we have made a number of additional base enums extensible.

Let's take enum TMSFeeType as an example. Assume we have made it extensible in X++. That means that in our C# project, where we use this enum, we will no longer be able to access it from Dynamics.AX.Application namespace by name, like so:

switch (accessorialFeeType)
{
    case TMSFeeType.Flat:
        // Do something
        break;

    case TMSFeeType.PerUOM:
        // Do something else
        break;
    // etc.
}

If you navigate to its definition, you will notice that the enum declaration is empty:

namespace Dynamics.AX.Application
{
    public enum TMSFeeType
    {
    }
}

The proper way to use the enum that is extensible is to reference the above mentioned class suffixed with _Values, which lives in the Dynamics.AX.Application.ExtensibleEnumValues namespace, like so:

if (accessorialFeeType == TMSFeeType_Values.Flat)
{
    // Do something
}
else if (accessorialFeeType == TMSFeeType_Values.PerUOM)
{
    // Do something else
}

Note: Because the values are now determined at runtime by going to the AOS and asking for the correct integer value of this enum, they cannot be used in a switch/case block, which expects constant expressions known at compile-time.

What's next

Obviously, this situation is not great. 

Let's hope that Microsoft will think of a good way to address this going forward.

Question to you

That leads to a question - how many of you actually have .NET libraries relying on application code in Dynamics 365 for Operations and might be impacted by us making some of the enums extensible in the next major release?