Hunting 'Mr. X' … uhhh … '0x1C'

The other day I was assigned the job of installing a couple of NAV 2009 (web) servers, which included setting up delegations. Lucky me. [:(]

I ran into an issue that kept me busy for almost 2 days which eventually ended in a, as so often, very simple fix. [:^)]

Of course to prevent you and myself from having to do this 2 days endeavor I'll share my findings below. If you can't hold your nerves just jump to my Long Story Short.

But maybe even more I am writing this story as an homage to all my fellow NAV pros that have consciously and, all but one, unconsciously contributed to the solution. Every workday I am grateful to all of you that share your findings on fora like mibuso, dynamics user and Microsoft Dynamics NAV Community even if you pose them as questions.

So here we go with my Short Story Long.

Short Story Long – A Homage

Here I am, setting up NAV 2009 (web) servers and delegations. Typically a task that I do not perform on a regular base and which often is a nuisance due to these delegations. But every time again I happen to get it working with valuable help of one of my IT colleagues, and so I am guaranteeing I will be the chosen the next time. [H]

BTW: needless to say that this all about a 3-box setup as described here.

This time the main goal was to get a NAV web server running to enable an external party to access our Dynamics back-end. Although we use web services heavily, our main installation is being accessed still by means of classic clients. But in the not too far future we will start running RTC and therefor, having installed the new web service, I wanted, in one go, to get the application service tiers running on our test and acceptance environments.

Easy said then done as it appeared.

Here starts the hunt for 'Mr. X'. Or actually '0x1C', as we will see. Even though we will not be using buses, taxis or the London Underground, we will hunt him down like the detectives in the Scotland Yard board game. We lay out the board and distribute the tokens and other accessories and confront the …

Initial Enigma

Where did 'Mr. X' go?

After having installed the right RTC and Service Tier builds, and configuring the service, starting up RTC threw the following error.

A first sign of live from Mr. X. Speaking Dutch, by the way.

ENU: The server net.tcp://<Server>:<ServicePort>/<ServerInstance>/Service is either unavailable or your connection has been lost. Do you want to attempt to reconnect?
[Note that I have replaced the server, port and instance names by generic terms.]

Clicking either Yes or No resulted in a follow-up message:

ENU: Lost connection with the server. The application is closed.

First thing …

… to check was whether the service was still running or not: it appeared to be running.

No sign of Mr. X. Where did he go?

Luckily I always install a full demo version on any NAV environment, be it development, test, acceptance or production. On the default port (7046) the application service for the demo database is running, while on another port (for example 7066) I can access development, test, acceptance or production database. Would I be able to access it? In order to check that the …

Second thing …

… was to change the ClientUserSettings.config so that my RTC would be looking at the demo service tier (<Server>:7046/DynamicsNAV/Service) and run RTC.

Splash screen, as always slow, keeping me waiting … and … keeping me waiting … and … keeping me waiting … and …

Alive and kicking! Clear that delegations were not blurring things. Although I still had to check whether using a fully qualified DNS would also work OK.

Entering the fully qualified DNS in the Select Server window and pressing TAB showed the available companies in the demo database. Clicking OK resulted in a perfectly normal start-up of my RTC.

Would I be able to do the same with the fully qualified DNS for our test database?

And get closer to my Mr. X?

Didn't halter too long and did the …

Third thing ….

…, i.e. entering the fully qualified DNS for our test database and pressing TAB. Wow, connected to the service tier!

Mr. X, we feel somewhat close to you, but are we? And how to get closer now that I ran out of my first options. What were you actually saying?

Clicking however, threw the next error.

The server net.tcp://<Server>:<ServicePort>/<ServerInstance>/Service is either unavailable or your connection has been lost. Do you want to attempt to reconnect?

Help can be quite close …

… if you can find it. Well, let's at least attempt to. Google is your Friend, isn't he? Straight around the corner.

Google, let's frame Mr. X.

Google: http://dynamicsuser.net/forums/p/39925/204742.aspx

What did you say? Trying to solve one riddle, getting another in turn? [:^)]

Google: The server net.tcp://<Server>:<ServicePort>/<ServerInstance>/Service is either unavailable or your connection has been lost. Do you want to attempt to reconnect?

OK, makes more sense. Let's click on it. Aha, my dynamicsuser friend Manish found a solution to this error. Indeed my default RTC profile was set to PAG50175 (part of our customization), so I changed it to one of the standard pages: PAG9006. No luck. Still getting the same error. To be sure I even removed all the records in the User Personalization (TAB2000000073), but to no avail.

Google: RTC Error “The server <address> is currently not available”

Next hint, you mean?

Well, glad you lead me to my dear fellow MVP Arend-Jan Kaufmann. Always providing valuable, and often technically detailed information. But no, this is not the issue. I know by heart we have the same build running on all machines. OK, I check it once more …

… maybe I accidentally run into Mr. X. [H]

Back again, Google, and no, found nothing valuable.

And surely no Mr. X.

Got some more riddles for me?

Error: '┴' invalid character

Nope, doesn't seem what I am looking for. No clear cut fix.

In hindsight it was more or less what I was looking for. I was actually sitting next to Mr. X!

It surely pointed out me to have a look at the Event Log; as with the next as with Google's next hint:

metadata error

As with the previous hint it was too much info in that error message. However this had some clear suggestions which at least I could give a try haven't succeeded so far. But before continuing my search I decided to ask Arend-Jan for help by sending him a mail with the event log file. If I wouldn't find a solution over time he might.

I recon meanwhile Mr. X was laughing in his sleeve making sure I could not hear it.

So based on Google's last hint decided to remove all records form the Object Metadata table and recompile all objects. I guess I was somewhat overreacting by removing all records. Just tired of searching and really getting closer to a solution. Starting RTC again I was welcomed by the next error of which I totally forgot to make a screen shot:

—————————
Object of type Table with id 2000000045 could not be found.
—————————

Mr. X has left the room!

Ouch, I am drifting away.

Half an hour later I found myself back on track, again getting some help from another of my fellow MVPs, Saurav Dhyani, on rebuilding the metadata. Don't know why, but this however lead to a Change Listener issue. After solving this and again compiling all objects … I found myself having a deja vu: I was looking at the initial error again. [:@]

This suggestion from again a fellow MVP, Natalie Karolak, also didn't help me out either unfortunately even though it might prove valuable in the future.

And then I received an answer form Arend-Jan actually not doing much more then focusing on the invalid character as already mentioned above.

Long Story Short

Arend-Jan Kauffmann out pointed that this was the crucial clue in the Event Log. He wrote:

Apparently the XML data, that is generated by the NAV server and sent to the RTC, contains a character that is not supported by the RTC. Probably some strange character in a caption or so. Or something else in the data.

Actually seeing our Mr. X almost straight in the eyes, but at the other site of the river Thames.

Within reach, but not ready to grab.

Mr. X, or, as mentioned before, Mr. 0x1C, being an invalid XML character, made the start-up sequence of the sequence of the RTC stop, resulting in this useless message:

The server net.tcp://<Server>:<ServicePort>/<ServerInstance>/Service is either unavailable or your connection has been lost. Do you want to attempt to reconnect?

Mr. 0x1C was clearly misleading us.

Fix

But still needing to find Mr. X.

How was I at long last able to grab him "by the collar"?

Using the Visual Studio Find in Files feature with the following regular expression [\x00-\x08\x0B\x0C\x0E-\x1F] being the illegal XML characters one object file popped up: MEN90.

It thus eventually turned out that Mr. 0x1C had been unconsciously and unintended placed in the Name property of one of the MenuSuite items the week before. Waiting to get out and get me busy for 2 days trying to trace him down … by talking plainly nonsense.

A big thanx to all my fellow NAV pros that have consciously and unconsciously contributed to the solution. Of course one special thanx to Arend-Jan.

Hope you can profit from my findings as already one of you did.

Note

Oh, one note to be made before I go: the most time consuming part of this endeavor was having to restart the RTC over and over again. Well actually not just restarting but needing to revert the ClientUserSettings.config … Can't this be made somewhat simpler, MS?