[Development tutorial] Thoughts on test automation + more sample tests using ATL

I recently had an interesting discussion around testing and specifically Microsoft testing efforts and frameworks, namely, ATL. If you are not familiar with this framework - you've been living under a rock. Get out from under there and go read some articles about it, like the one below

https://kashperuk.blogspot.com/2019/05/development-tutorial-sample-test-tips.html

The discussion revolved around the complexity and costs of adding and maintaining tests. The partner also questioned how often we add tests, and if we can share more samples of such. You can also see his comment with a more concrete ask in the blog post above. 

Here's a partial quote:

Partners need more example for this functionality usage. Is it possible that you provide real tests for some real issues (I just took several bugs from the latest PU Warehouse and Transportation Management)

533573 Creating a sales order’s work without a pick location is not updating the on-hand figures correctly.

535311 Cannot do material consumption from mobile device for shelf life items

536899 The container contents report does not show an error if shipment does not have a dropoff address specified

537772 Cluster profile could not be found please check your configuration Error is blocking Purchase Order Receiving

If you share them in a new blog post as examples that will be a great discussion point

So in this post, I'd like to share 2 things:

1. Generally speaking, we at Microsoft, or rather, more specifically, we in the Warehouse management team, add one or more tests for every single customer reported issue. 

This obviously adds a cost to each bug fix we do, but with this we get a much more reliable release schedule and on-going confidence in our product offering, and have very few regressions despite heavy code churn.

The point I want to make here is - if you are an ISV, and you have not covered your solution with test automation, you should do it now, and reap the benefits. 

If you are a partner getting paid by the hour doing a specific customer implementation, it gets more tricky. However, I believe (I know) it is in your and your customer's interest to justify the cost of adding test automation.

Very frequently today, even with some of our larger customers, we hear test cycles of 1-2 months before taking a newer standard release. That puts a significant on-going cost on the customer, as well as stress, as we at Microsoft keep pushing everyone to be current. This could have been mostly alleviated with test automation

ATL is a framework which allows to write robust (and reasonably fast) test automation with a low cost investment, all things considered. 

2. We do not ship our test automation library. The code is not written with shipping in mind, so it's not necessarily "Microsoft production quality". The value for customers is also marginal. 

But, to answer the ask from the quote above directly, here are the tests added for the 4 issues listed. I did not include the class/test setup logic, nor did I verify these tests are able to run on your specific dataset (internally we have tests that are both dataset dependent and independent). So take them for what they are - mere samples that prove the point - all of these are rather easy to understand when reading, and are rather easy to write, if you know the scenario you want to automate. For all or most of the standard functionality there are already ATL "wrappers" ready to be used.

• 533573 Creating a sales order’s work without a pick location is not updating the on-hand figures correctly.

	[SysTestCheckInTest]
    public void soRelease_StopWorkOnLocDirFailureFalse_onHandCorrectWorkCompleted()
    {
		// Given
        const InventQty QtyOnReceiptLocation = 100;
        const InventQty SalesQuantity = 1;

        ttsbegin;

        var locDirFailure = whs.locationDirectiveFailures().sales();
        locDirFailure.LocDirFailWork = NoYes::No;
        locDirFailure.update();

        var locProfile = whs.locationProfiles().receipt();
        locProfile.setLPControlled(false).save();

        var receiptLocation = whs.locations(warehouse).create(locProfile);

        // Delete existing sales pick location directive to ensure no impact to the one created.
        var locDir = AtlEntityWHSLocationDirective::find('24 SO Pick', WHSWorkType::Pick, WHSWorkTransType::Sales, warehouse);
        locDir.delete();

        whs.locationDirectives().salesPick().setLocationRange(pickLocation).moveToTop();

        onhand.adjust().forItem(item).forInventDims([receiptLocation]).addOnHandQty(QtyOnReceiptLocation);

        salesOrder.addLine().setItem(item).setInventDims([warehouse]).setQuantity(SalesQuantity).setAutoReservation().save();
        ttscommit;

		// When
        salesOrder.releaseToWarehouse();

		// Then
        var work = salesOrder.work().singleEntity();

        work.lines().withTypePick().withLocationId('').assertSingle();
		
        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
			.forReferenceId(work.parmWorkId()).assertCount(0);

		onHand.assertExpectedOnHand(
            onHand.spec().forItem(item).withInventDims([warehouse]).withAvailTotal(QtyOnReceiptLocation - SalesQuantity));

        var soPicking = rf.login().salesOrderPicking()
			.setWork(work)
            .setLocation(receiptLocation);

        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
            .forReferenceId(work.parmWorkId()).withStatusIssue(StatusIssue::ReservPhysical).assertSingle();
        invent.trans().query().forItem(item).forReferenceCategory(InventTransType::WHSWork)
            .forReferenceId(work.parmWorkId()).withStatusReceipt(StatusReceipt::Ordered).assertSingle();

        soPicking
			.confirmPick()
			.confirmPut()
			.assertWorkCompleted();
    }  
  

• 535311 Cannot do material consumption from mobile device for shelf life items

      [SysTestCheckInTest,
        SysTestCaseAutomaticNumberSequences,
        SysTestFeatureDependency(classStr(WHSProdMaterialConsumptionJournalBOMBatchExpiryDateFeature), true)]
    public void materialConsumption_BatchIsExpiredBySchedDate_JournalLinesCreated()
    {
        const Qty BomLineQty = 1;
        const Qty ProductionOrderQty = 1;

        ttsbegin;

        // Given

        InventModelGroup fefoGroup = invent.modelGroups().fefoWithExpiryCriteriaBuilder().save().record();
        fefoGroup.PickingListBatchExpirationDateValidationRule = WHSPickingListBatchExpirationDateValidation::ReservationDate;
        fefoGroup.update();

        InventTable rawMaterial = items.whsBatchAboveBuilder()
            .setItemModelGroup(fefoGroup)
            .setPdsShelfLife(PdsShelfLife)
            .create();

        InventBatch validBatch = invent.batches().create(rawMaterial);
        validBatch.setExpiryDate(DateTimeUtil::getToday(DateTimeUtil::getUserPreferredTimeZone()) + PdsShelfLife).save();

        onhand.adjust().forItem(rawMaterial).forInventDims([floor, lp1, validBatch]).addOnHandQty(RMItemQty);

        InventTable finishedGood = items.whsBatchBelowBuilder()
            .setItemModelGroup(fefoGroup)
            .setPdsShelfLife(PdsShelfLife)
            .create();

        AtlEntityBillOfMaterials billOfMaterials = this.createBOMAndBOMVersion(rawMaterial, BomLineQty, finishedGood);
        AtlEntityProductionOrder productionOrder = this.createProductionOrder(finishedGood, billOfMaterials, ProdReservation::None, ProductionOrderQty);
        productionOrder.estimate().execute();
        
        productionOrder.scheduleOperation().setSchedDirection(ProdSchedDirection::ForwardFromSchedDate).setFiniteCapacity(false)
            .setFiniteMaterial(false).setSchedDate(DateTimeUtil::getToday(DateTimeUtil::getUserPreferredTimeZone()) + PdsShelfLife + PdsShelfLife).execute();

        productionOrder.start().setAutoBOMConsumption(BOMAutoConsump::Never)
            .setPostPickingList(false).execute();

        ttscommit;

        AtlWhsWorkExecuteProdMaterialConsumption materialConsumption = this.createMaterialConsumptionWithBatch(productionOrder, floor, rawMaterial, validBatch);
        
        materialConsumption.setQuantityToConsume(ProductionOrderQty);

        // When and Then
        materialConsumption.assertMessageOnClickOk("@WAX:MaterialConsumptionJournalLineCreated");

        materialConsumption.assertMessageOnDone(
            strFmt(
                "@WAX:MaterialConsumptionJournalPosted",
                productionOrder.pickingLists().withJournalType(ProdJournalType::Picklist).singleEntity().parmJournalId()));
    }
  

• 536899 The container contents report does not show an error if shipment does not have a dropoff address specified (Not using ATL, as this was an old existing test. More than 1 test was added / modified here)

      [SysTestCheckInTest]
    public void preRunValidate()
    {
        // arrange
        InventSiteLogisticsLocation inventSiteLogisticsLocation;
        delete_from inventSiteLogisticsLocation;

        Args args = new Args(formStr(WHSContainerTable));
        FormRun formRun = classfactory.formRunClass(args);
        formRun.init();

        FormDataSource containerTableDS = formRun.dataSource(tableStr(WHSContainerTable));
        containerTableDS.executeQuery();
        containerTableDS.markRecord(containerTable, true);

        args = new Args();
        args.caller(formRun);
        args.record(containerTable);

        WHSContainerContentsControllerTestable controller = new WHSContainerContentsControllerTestable();
        controller.parmArgs(args);

        Query query = new Query(queryStr(WHSContainerContents));
        controller.setRanges(query);

        Map queryContract = new Map(Types::String,Types::Class);
        queryContract.insert('Query',query);

        SrsReportDataContract srsReportDataContract = new SrsReportDataContract();
        srsReportDataContract.parmQueryContracts(queryContract);
        controller.parmReportContract(srsReportDataContract);

        // act
        container validateResults = controller.preRunValidate();

        // assert
        this.assertEquals(validateResults, [SrsReportPreRunState::Error, "@WAX:WHSContainerContentsReportMissingPrimaryAddress"], 'Unexpected value in validateResults.');
    }
  

• 537772 Cluster profile could not be found please check your configuration Error is blocking Purchase Order Receiving

      [SysTestCheckInTest]
    public void poReceive_QualityOrderAndAssignClusterEnabled_WorkAssignedToCluster()
    {
        const Qty PurchOrderQuantity = 2;
        ttsbegin;
        // Given
        whs.warehouses().ensureQualityManagementEnabled(warehouse);
        invent.qualityOrders().ensureCanCreate();
        whs.locationDirectives().qualityItemSampling(whs.locations(warehouse).stage().wMSLocationId, warehouse);
        whs.workTemplates().qualityItemSampling();
        invent.qualityAssociations().defaultBuilder()
            .setOrderType(InventTestReferenceType::Purch)
            .setApplicableWarehouseType(WHSApplicableWarehouseType::QualityManagementOnlyEnabled)
            .setItemRelation(whsItem.ItemId)
            .setSiteId(warehouse.InventSiteId)
            .setDocumentType(InventTestDocumentStatus::Registration)
            .setShowInfo(NoYes::Yes)
            .setAcceptableQualityLevel(100)
            .setQualityProcessingPolicy(WHSQualityProcessingPolicy::CreateQualityOrder)
            .setItemSampling(invent.itemSamplings().onePiece())
            .getResult();
        clusterProfile = whs.clusterProfiles()
            .putawayClusterManualAssignAtReceipt(WHSWorkTransType::Purch, whs.workTemplates().purchaseReceipt().parmWorkTemplateCode());
        ttscommit;
            
        this.addPurchaseOrderLine(whsItem, PurchOrderQuantity);

        // When
        var poItemRecv = rf.login().purchaseOrderItemReceiving()
            .setPurchaseOrder(purchOrder)
            .setItem(whsItem)
            .setQuantity(PurchOrderQuantity);

        AtlQueryWHSWork workQuery = purchOrder.work().withClusterProfileId(clusterProfile.parmProfileId());
                
        // Then
        poItemRecv.assertErrorOnConfirmUnitSelectionWithWorkQuery(workQuery, clusterProfile.parmProfileId(), 'Wrong message');
        invent.qualityOrders().query().forPurchaseOrder(purchOrder.record()).assertCount(1, 'Should have 1 quality order');
    }
  

Hope this helps. Do reach out to us with feedback / suggestions.

Do make use of CDE (Community Driven Engineering) to submit additions / bug fixes. We do expect ATL tests with those fixes.