MQTT Connect / Disconnect

[BobO]

With the broker shut down, I would expect the MQTTPUB to fail and jump to the error stage. If it is important to publish everything without fail, you just need to jump back to the publish stage until it succeeds, rather than jumping back to the FIFOUNLOAD. If it is claiming success without completing the publish, that?s a bug.

I?ll play with it more.

[Andrew S]

If it is claiming success without completing the publish, that?s a bug.

That appears to be what I've seen. It doesn't happen every time but I believe I've seen it multiple times.

[BobO]

I tested two ways: 1) pulling the cable, but leaving everything on both sides running, and 2) killing the broker.

When killing the broker, I occasionally see a bit of weirdness on mosquitto's side, where he is publishing more than once back to the PLC (as part of the subscribe), suggesting he saw the publishes from the PLC, but he doesn't actually show the publishes from the PLC and the data sent to the PLC is old. That would appear to be mosquitto bug.

When leaving the broker running and pulling the cable, the PLC is giving me a high-five on the publish as long as the connection hasn't yet timed out. It starts returning errors when the connection closes. That's a bug on our side. I'll look into it.

[BobO]

That appears to be what I've seen. It doesn't happen every time but I believe I've seen it multiple times.

I figured out my issue. It was set to QoS 0, which doesn't have any form of error checking. the publish packets get dropped into the socket queue, and even though they don't really get sent, the application layer doesn't really know that. When I changed it to QoS 1, it is correctly reporting the error, and the retry code in my PLC program got the result through.

[Andrew S]

That's interesting. My QoS setting is already set to 1 though. I'll do some more testing when I have a moment.

[BobO]

That's interesting. My QoS setting is already set to 1 though. I'll do some more testing when I have a moment.

I'm not saying there isn't an issue, but I haven't been able to demonstrate a problem once I got the QoS correct. There was definitely an issue with mosquitto, but it only showed when I shut it down and started it.

How are you handling the retry in your FIFO publishing program? All I'm doing is logging the retry in the error stage and jumping right back to the publish stage. It doesn't go back to the FIFO fetch stage until it successfully completes.

[Andrew S]

I'm doing something similar. I pop a record from the FIFO and store it in a UDT Structure. Then I attempt to PUBLISH it. If it fails I jump to a new stage, wait a minute, and then jump right back to the PUBLISH stage and attempt to resend. It keeps doing this until it successfully sends it. On success it checks the FIFO again and everything moves forward.

The failure scenario I've been testing has been against a local dockerised Mosquitto broker that I kill to simulate a network/server failure. Then I dump some data into the FIFO while the broker's down, and I see what happens when the broker restarts.

The issue seems to be that the PUBLISH returns success before it's actually done processing (the same issue you mentioned). I'll see if I can grab some logs that include the lost-data scenario.

[BobO]

If you are restarting mosquitto, it could be their bug. What I saw in their verbose dump was them publish to the PLC (it was subscribed to the same topic I'm publishing) in response to the PLC publishing to mosquitto, but the log didn't show mosquitto ever receiving the publish from the PLC to start with. That can't be, because they wouldn't publish multiple times due to a single subscription, and it was showing more than one. It appears that they are getting tripped up during a pregnant moment at startup.

When it works correctly, I see both the publish from the PLC to mosquitto, and mosquito publishing back to the PLC's subscription. When it fails, I just see the effect, and not the cause.

If you aren't already, run mosquitto with -v.

[Andrew S]

Interesting. I'll see if it does the same thing against VerneMQ (probably later this week).

[BobO]

Messed with it a bit more today. I'm gonna walk some of that back. I do see cases where a publish isn't happening even though it should be.

And now I can't duplicate the behavior I was seeing from mosquitto. Weird.

[Andrew S]

Weird.

Welcome to my life ;-)