News:

  • February 05, 2023, 07:06:25 PM

Login with username, password and session length

Author Topic: H2-ECOM100 performance gradually degrading  (Read 2500 times)

jeffcd

  • Newbie
  • *
  • Posts: 7
H2-ECOM100 performance gradually degrading
« on: March 01, 2019, 11:29:04 AM »
Hi All,

I have a DL260 PLC with an ECOM100, running a wastewater plant.
We recently upgraded the SCADA system to Ignition (Inductive Automation) and the SCADA is communicating to the DL260 via Modbus TCP.  Ignition is only exchanging 250 tags with the DL260.
The ECOM100 is also connected to 2 C-More HMI's, a remote rack with another DL260 with approx 20 I/O points and a Red Lion Datastation Plus that's reading a couple words of data and sending to an AB PLC.

Since the Ignition SCADA and the Datastation were installed, we've been having an issue where the network comms seem to get progressively worse over time.  It was so bad at one point, the operators had to power cycle the control panel every morning, because the data on the C-More's was stale/frozen.

The Ignition SCADA has a diagnostics page that shows the "Load Factor" of each device.
When we starting having the network issues, I noticed the Load Factor was over 200%.  (the load factor takes into account things like response time and requests/sec to determine if the device is keeping up with the tag requests).  I did some optimizing on the Ignition side of things and slowed down the polling rate to 2.5sec and managed to get the Load Factor down to around 100%.

Then I updated the firmware on the DL260 from v2.6 to 2.7 and the Load Factor dropped to 25% and that was with a poll rate of 1sec!
I thought I had solved it at that point, but about a week later the Load Factor had crept up over 100% again.
So then I updated the firmware on the ECOM100 (it was only one version behind) and again the Load Factor dropped to around 35%, but it's only been a day now and it's already increased by 15%.

So I'm not sure if the firmware updates themselves are actually causing the temporary improvements or if it's simply because I'm clearing some sort of memory buffer during the update.
The module seems capable of handling the speed, because after the updates the response time was around 45ms compared to before when it was in the 250ms range.

I've called both Automation Direct and spent many hours on the phone with Ignition's tech support, but so far we haven't solved the issue.

I've attached a Wireshark capture if it's any help.  The ECOM100 is 192.168.1.10 and the Ignition SCADA server is 192.168.1.81.

Could it be something to do with the requests from the Ignition software that is causing the problem?
Any insights here would be greatly appreciated, because I'm running out of ideas!

Thanks,

Jeff

ADC Product Engineer

  • Hero Member
  • *****
  • Posts: 270
Re: H2-ECOM100 performance gradually degrading
« Reply #1 on: March 03, 2019, 12:26:05 PM »
A few things come to mind with this type of scenario.  Looking at your Wireshark, I see where you start to get retries and spurious transmissions after a while.  This causes the ECOM to get backed up and take longer to respond which causes a snowball type of effect.  If you go into NetEdit and look at the statistics, I think you should see this shown.

Things to check:

1. Increase poll time for each device.  Mainly the C-more's, by default they are set to go as fast as possible.  If you are hitting as hard as you can with every device, there is little to no time for the ECOM to respond.  It only has time at the beginning of scan to place and retrieve data from the CPU.

2. If #1 doesn't do it, check your system for noise.  The first place to start is with cable routing.  All cables should be placed as far away from AC power sources as possible.  Also, if you have a switch in the panel, try putting it on it's own power supply.  I have seen quite a number of cases where this sort of behavior is caused by the switch inducing noise from coils, motors, etc into the Ethernet system.

3.  Check power and grounding for the entire system.  Make sure everything that has a ground spot is grounded.  Make sure that your coils, relays, solenoids, etc are properly suppressed.  Drop a power monitor on and make sure you aren't browning out at some point during the day.

Controls Guy

  • Internal Dev
  • Hero Member
  • ****
  • Posts: 3478
  • Darth Ladder
Re: H2-ECOM100 performance gradually degrading
« Reply #2 on: March 03, 2019, 02:16:10 PM »
Maybe reorganize some of the logic to slice over consecutive scans like a Do-More.  If the PLC is handling comm synchronously, more scans should help.   Maybe even swap out to a Do-More.
I retract my earlier statement that half of all politicians are crooks.  Half of all politicians are NOT crooks.  There.

jeffcd

  • Newbie
  • *
  • Posts: 7
Re: H2-ECOM100 performance gradually degrading
« Reply #3 on: March 04, 2019, 09:20:58 AM »
Thanks to all for the replies.

@ADC, I had the cmore's set for a poll rate of 1 sec.  I've now changed them both to 2 sec, but it hasn't made much of an improvement.
I will look into items 2 and 3 in a couple of weeks when I'm back on-site.
BTW, I checked NetEdit and all the stats show zero.
The Redlion Datastation Plus was also set to poll as fast as it could, so I've currently changed it to poll at 250ms.

Before this weekend I changed the SCADA poll rate from 1sec to 2sec and the load factor dropped to 99%.  Checked again this morning and it seems to be holding at 99%.
So it seems that there is a "saturation point" where if the poll rate is too fast it causes the snowball effect you mentioned, where the load factor just continually gets worse, but if I set the SCADA poll rate at a slower value, the ECOM can keep up and maintain it.

@Controls Guy, thank you for the suggestion, but can you explain what you mean by reorganizing my logic to slice of consecutive scans?  I'm not sure that I understand that one.
I may suggest to the customer to swap to a Do-More at some point, but they just went through a big upgrade and I don't think they'll be open to spending more money to change something that's been working fine for the last 11 years.  I'd have to re-write and test almost 1000 rungs of code (although I understand that it might be a little easier in the do-more environment).

franji1

  • Bit Weenie
  • Host Moderator
  • Hero Member
  • *****
  • Posts: 3468
    • Host Engineering
Re: H2-ECOM100 performance gradually degrading
« Reply #4 on: March 04, 2019, 09:57:53 AM »
To get more comms through per second, if you reduce your PLC scan time, then you can get more packets processed per second (PLC only does 1 to 4 packets per PLC scan, so the PLC scan time directly affects the packet rate per second).

If your PLC scan was 100ms, if you were able to reduce it to 50ms, that could possibly double your comm rate (theoretically; it may not actually double, but it will greatly improve)

However, if you happen to have remote devices that are polling "as fast as possible", that would just make them send more packets per second since the PLC would be responding faster.  That just exacerbates the problem. 

Hence, setting the poll rate of your remote devices to "fast enough", not "as fast as possible", THEN reduce the PLC scan time, that will help the PLC "breathe", especially when you connect up with DirectSOFT (yet another "device" that polls).

jeffcd

  • Newbie
  • *
  • Posts: 7
Re: H2-ECOM100 performance gradually degrading
« Reply #5 on: March 04, 2019, 11:01:32 AM »
@franji - Thank you for the reply, I will take that into account and see if there's some potential to optimize the code and reduce the scan time a little.

I've seen your name on these forums for many years and your posts have helped me (and I'm sure thousands of others) through a number of issues, so thank you for all your contributions!

jeffcd

  • Newbie
  • *
  • Posts: 7
Re: H2-ECOM100 performance gradually degrading
« Reply #6 on: March 04, 2019, 11:05:39 AM »
By the way, I forgot to mention that my current scan time on the DL260 is 60ms.

Controls Guy

  • Internal Dev
  • Hero Member
  • ****
  • Posts: 3478
  • Darth Ladder
Re: H2-ECOM100 performance gradually degrading
« Reply #7 on: March 04, 2019, 11:48:55 AM »
@Controls Guy, thank you for the suggestion, but can you explain what you mean by reorganizing my logic to slice of consecutive scans?  I'm not sure that I understand that one.

There is probably a lot of the logic that doesn't need to be run each and every scan.  Do-More has blocks of code that called Programs and Tasks that are, first, selectively executed, and second, can be interrupted before completion each scan once they've run for so many microseconds.  That allows you to execute your code no faster or more often than needed to do it's job, freeing up resources to execute more time-sensitive code more promptly, or to reduce scan time.

You can do the same thing in your DL260 by selectively executing SBR's or JMP's.  Plus, if you execute half the code each scan, you'll cut the logic portion of the scan by 50%, so even after you add back the new comms time between the increased number of scans, you won't have doubled the effective scan time of any particular rung of code.

Quote
I may suggest to the customer to swap to a Do-More at some point, but they just went through a big upgrade and I don't think they'll be open to spending more money to change something that's been working fine for the last 11 years.  I'd have to re-write and test almost 1000 rungs of code (although I understand that it might be a little easier in the do-more environment).

Well, it's not the same thing now that's been working fine for 11 years.  It's not really logical for them to say "It's been running the equipment OK for 11 years so we can add new comm load without limitation and be surprised if it doesn't still work."

You might also consider adding a second ECOM, though if it's scan time limiting the bandwidth, that's not going to help.  You could also add a comms concentrator in the form of a new PLC or maybe you can do it with some of the hardware you have there now.  Just have one device poll the PLC for all the registers needed by any one device (realtime writes like button pushes should probably still go direct to the PLC) and have everyone else talk to that device.  Fewer requests for larger blocks (and sequenced by this concentrator device) will be more efficient for the PLC.

Finally, I agree with ADC PE and Franji1 that no clients should be set to poll as fast as possible (probably ever, but certainly in this scenario).
I retract my earlier statement that half of all politicians are crooks.  Half of all politicians are NOT crooks.  There.

jeffcd

  • Newbie
  • *
  • Posts: 7
Re: H2-ECOM100 performance gradually degrading
« Reply #8 on: March 04, 2019, 01:06:37 PM »
Quote
There is probably a lot of the logic that doesn't need to be run each and every scan.  Do-More has blocks of code that called Programs and Tasks that are, first, selectively executed, and second, can be interrupted before completion each scan once they've run for so many microseconds.  That allows you to execute your code no faster or more often than needed to do it's job, freeing up resources to execute more time-sensitive code more promptly, or to reduce scan time.

Thank you for all the tips!  As a quick test, I added a GOTO/LBL instruction to only execute 80% of the ladder logic every 100ms.
That made a massive improvement!  This is definitely a workable solution, since this is a process control system that doesn't need to do anything at lightspeed.

I "inherited" this code years ago and have been trying to maintain and improve it little by little over the years, but it is one long program with no subroutines or JMP's, so there's room for improvement for sure.

One more question: Will the timers still run accurately in the chunks of code that I'm only running every 100ms now?  For example, am I now messing up my delay timers or 1 sec pulses that run my flow totalizers?

Thanks again for all the help!

franji1

  • Bit Weenie
  • Host Moderator
  • Hero Member
  • *****
  • Posts: 3468
    • Host Engineering
Re: H2-ECOM100 performance gradually degrading
« Reply #9 on: March 04, 2019, 01:30:25 PM »
One more question: Will the timers still run accurately in the chunks of code that I'm only running every 100ms now?  For example, am I now messing up my delay timers or 1 sec pulses that run my flow totalizers?
Anything time based MUST run every scan.  Drums, Timers, etc.

jeffcd

  • Newbie
  • *
  • Posts: 7
Re: H2-ECOM100 performance gradually degrading
« Reply #10 on: March 04, 2019, 01:39:58 PM »
Quote
Anything time based MUST run every scan.  Drums, Timers, etc.

Ok, good to know.  That changes things a bit... I'll have to use a scalpel instead of an axe... but it's doable.

Thanks.

Controls Guy

  • Internal Dev
  • Hero Member
  • ****
  • Posts: 3478
  • Darth Ladder
Re: H2-ECOM100 performance gradually degrading
« Reply #11 on: March 04, 2019, 01:42:16 PM »
One more question: Will the timers still run accurately in the chunks of code that I'm only running every 100ms now?  For example, am I now messing up my delay timers or 1 sec pulses that run my flow totalizers?

Yeah, timers assume they're being executed every scan.  Three possible workarounds:

1) Put just the timers back in the critical every-scan section.
2) Change to the internal one second bit (SP4?)
3) Schedule stuff rather than run timers, either off the PLC wall clock or a master 32-bit timer in the every-scan code and use compares with the ACC of that timer.
I retract my earlier statement that half of all politicians are crooks.  Half of all politicians are NOT crooks.  There.

Controls Guy

  • Internal Dev
  • Hero Member
  • ****
  • Posts: 3478
  • Darth Ladder
Re: H2-ECOM100 performance gradually degrading
« Reply #12 on: March 04, 2019, 01:49:11 PM »
Oh, and you can also count pulses of the internal flasher bits so long as your total round-robin effective scan time is always under half the period of that specific flasher.  In effect, that creates a timer.
I retract my earlier statement that half of all politicians are crooks.  Half of all politicians are NOT crooks.  There.

jeffcd

  • Newbie
  • *
  • Posts: 7
Re: H2-ECOM100 performance gradually degrading
« Reply #13 on: March 04, 2019, 01:52:04 PM »
Great tips!
At least now I have a path forward to a solution.
This issue has been eating me for a while now.

Cheers!