You want to save bad packets and
review them to the bit level?
I hear a lot of ideas and I love most of them as they lead
to better products, better statistics and information. This was recently asked
of me - I want to capture and save every packet – good and bad/invalided
packets for later review. In reality to capture every Bit, Nibble and Octet
including the InterFrame Gap!
Now this is a huge task!
By – Tim O’Neill –
The Oldcommguy™
Special Thanks to Maureen
Rozenhart of Anue Systems for her support!
Some initial thoughts!
I agree that in today’s world of Compliance, Security..Etc
one should and needs to save lots of data, maybe even all the data for a long
time, maybe even for years.
I do not agree with saving the InterFrame Gap (IFG) and bad
packets. I believe that one should just be able to save the bad frame
statistics and only save the real data frames for later review, evidence and
loss mitigation.
Why would one save
the Bad Frame Statistics? The only reason for reviewing Bad Frames, in
my humble opinion, is for hardware developers that want to see how their
firmware and drivers are functioning in a real world environment. There are
certain security reasons where a hardware analyzer is needed but this is a
subject for future discussions.
Everyone needs to do baselining on a regular basis to make
sure that the application layers (layer 4 up) are running on a solid
foundation. The Foundation of a data network are layers 3 and down. If one is
getting lots of bad frames and thus lots of retransmissions the network and
thus the application efficiency is going to be lower. Bad Frame counts and Retransmissions are only
a few of the important items in a Base Line Analysis but are essential in any
analysis.
It is the old sage – Do not try to build a 5 story network
on a one story foundation. So Baselining is an essential element for day to day and
maybe even hour to hour in a good monitoring plan.
What is needed -
So
what tools do you need to carry out this VERY demanding task of Capturing every bit, being able
to Review in a filtered and
focused format (with relevant and comprehensive statistics) and be able to Analyze and Look at all the data
at the bit level…..
1) Capturing large amounts of data for historical and diagnostic view – I
recommend the reasonably priced GigaStor from Network Instruments with its full
capture capability.
2) To Filter, Focus and Review the data - I recommend the Anue Systems
Net Tool Optimizer. It is reasonably priced and has a full and comprehensive
GUI that anyone can use effectively to look at any data characteristics.
3) To look at the data, every bit of it - I recommend the Absolute
Analysis Investigator. The AAI is a full bandwidth hardware based analyzer that
even sees the IFC and all the bad frames/bits, which we cover in more detail
below.
a. If one needs a higher level
focused analyzer I suggest the Network Instruments Observer.
There
are other products that can capture, filter and analyze but those I recommended
above are my favorites.
Please remember that to carry out this huge
task out one must have REAL Access to the network, through a TAP where
every bit, nibble and octet is passed regardless!
My
favorite TAPs are from Network Instruments and Network Critical, depending on
the final need.
What is a Bad Frame and What types
are there?
There
are different types of Bad Frames – Here are the main ones that should always
be included in a Baseline Analysis, alongside your data for full relevance.
Short Frame (Runt) – Any frame under 64 bytes
(512 bits).
Long Frame (Frame to long) – Usually any frame that
has more than 1,518 bytes or one that has exceeded the MTU (Maximum
Transmission Unit typically 1500 bytes in Ethernet).
Symbol Error – this is the number symbols (bytes) received that the system could
not decode correctly.
Fragments – This is where there is a count of the fragments
sent over the network.
Fragmentation occurs when a packet is too large to
be sent in a network link. When the packet/frame is too large, the original
packet is split into smaller packets that the network can handle; each fragment
contains enough information to allow the destination device to reassemble them
back to their original state/size at the conclusion of the transmission.
Fragmentation typically is initiated by a router. Once a packet is fragmented,
it will only be reassembled by/at the destination device.
Collisions – A Collision
is indicated when two devices detect/sense that the network is idle and try to
send packets at exactly the same time (within one round-trip delay). Because
only one device can transmit at a time, both devices must stop sending and
attempt to retransmit.
There are 4 collision types –
1. Single - packets sent after
one collision after wait period
2. Multiple - packets sent
after multiple collisions
3. Late - packets aborted
during sending because of collisions after 64 bytes
4. Excessive - packets not
sent because of too many collisions
Alignment Error - Frames are made up of a whole number of octets (FCS).
If a frame arrives and part of an octet is detected missing (number of bits not divisible by 8), this generates a Frame Check Sequence (FCS) error; this is labeled as a Alignment or Offset Error.
Frame Check Sequence error
(FCS) –
Similar to the CRC error, an FCS error indicates that the frame received did not have an integral number of octets. The CRC below will tell if any of the bits in the frame
have been corrupted in transit.
Errored Frame Check Sequence - A Cyclic redundancy Check (CRC) is the polynomial remainder calculated mathematical image of the Frame to confirm total frame integrity. It is a combination of FCS and Alignment errors. These errors below indicate how the packets were received with which error -
- A bad FCS and an
integral (whole) number of octets (FCS Errors)
- A bad FCS and a
non-integral (not divisible by 8) number of octets (Alignment Errors)
The CRC is a 4-byte polynomial mathematical code checksum remainder value. The CRC is calculated by the source
station. The CRC calculation is on all
the bits in the frame from the Destination MAC Address through the Pad fields
(that is, all fields except the preamble, start frame delimiter, and frame
check sequence). The source station stores the value in CRC field of the
transmitted frame. When the frame is received by the destination station, an
identical check is performed and compared to the transmitted value. If the
calculated value does not match the value in this field, the destination
station assumes an error has occurred during transmission. The frame is usually
discarded by the Network Interface card (NIC). If one has more than 3-5%
CRC/Alignment errors then this should be considered excessive and analysis
should be performed to identify the offending NIC/physical layer device.
Note - There are other Physical Layer issues – Jams, Late or Early
collisions, Dribbles, Jabbers, Giants…etc. that are not relevant to this discussion but can be important in lower layer analysis.
Most
modern data filtering devices can acquire and save these errors and more as
shown in the current Anue Systems Net Tool Optimizer GUI below.
Many
more statistical events can be captured and shared when the RFC 2665 dot 3 and
RFC 1757 MIBs are employed.
Anue's Net Tool Optimizer easy to use and powerful statistical view of errored frames!
Now if you did save every bit, nibble and octet and you need
to look at bad frames what do you need.
Well you need a hardware analyzer like the Absolute Analysis
Investigator and you need access to the full, data stream to see bad frames.
See picture below -
Absolute Analysis Investigator - Bit by Bit view including Bad CRC recognition and bad frame!
Let’s talk about saving all frames
for historical review/analysis –
Is
it practical to capture everything, every bit that flows – depending on one’s
needs and finances, yes it is possible.
If
one wanted to capture all the bits and frames in a small network with say five
10G links and one 1G link….my example
The minimum speed in this case is 1G and five 10G and they
want to capture for future analysis every packet (actually every bit including
InterFrame Gaps) – good and bad.
OK let’s start with capturing every packet – some basic
math…
One 1G circuits = 2 Gb/Second max to save
Five 10G circuits = 100 Gb/Second max to save
This will require at least five 10G full bandwidth of 20G
each capture devices and one 1G, actually 2G capability capture tool. This
calculates out to 102Gbits per second capture rate.
Most if not all capture devices will only capture good Ethernet
frames which are defined by the RFC as legal frames, this does not include
errored, short, bad CRC…etc frames (we discussed above) because almost every
NIC will drop the bad frames, especially if one is using SPAN or VACL access,
the Switch and Router will drop all the bad packets for you.
Now let’s say you can afford –
1) The cost of and support of five 10G and one 1G capture
engines with many Terabytes of storage. The size would depend on the time you
need to keep the data.
2) That you have the time to review 6120 Gb/minute or 367,200
Gb/hour or for 24 hours 8812.8 Tb or 1101.600 TB and at an average of 512B per
frame with 12B for IFG that is 2.102 trillion frames a day or let’s put that
into some easily handled numbers – It is Way toooooo MUCH!!!.
Most Network Analyst’s say that to review one packet takes
at least 30 seconds. So analyzing trillions of frames is absurd. So what does a
network engineer do to handle this mass of data and review it effective and in
a timely manner.
Ok, so this is where filtering devices come into play, like
the Anue Systems Net Tool Optimizer to filter out just what you need and forget
about trying to review every packet. One can save all the packets but with the
new advanced filtering technology one can easily review only the suspect data
or a specific application flow or conversation while using the most advance
MIB’s for keeping an eye on the fundamentals of your networks efficiency and
stability during the time in question.
Statistics and Focus is the Key where saving every bit,
nibble and bytes may not be the best this but may be a necessity for your
success.
In conclusion -
You can
capture everything on your network, analyze and review the data to the bit
level if you have the correct tools and the skills. Is Capturing everything needed in every network to the bit level, Yes
to a point based on one’s absolute need for review and potential for mitigation
of an incident. Do you need to see every bit and be able to review and analyze
to the bit level No that is not for everyone. Most Network Engineers analysis is
focused on good frames with statistics on the bad stuff is more than sufficient,
economical, and reasonable. Focusing on Good frames seems to keep most
engineers quite busy. Does everyone need TAP access to the Network, ABSOLUTELY
YES! Does everyone need Advanced Filtering technology to review with a focus;
Yes I think so because technology can work for you and focus on your analysis
and monitoring needs to make your analysis quicker, easier and more successful.
Be sure to
check out my favored vendors:
Anue Systems, Net Tool Optimizer at www.anuesystems.com
Absolute Analysis Investigator at www.absoluteanalysis.com
Network Instruments GigaStor, Observer and TAPs at www.networkinstruments.com
Network Critical TAP systems at www.networkcritical.com
Be sure to check out all our super sponsors - They all have great solutions -The ones above are the ones that I have used, that is all the difference and I am sure the others are super also!
Be sure to read my upcoming article on today’s easy to use
data filtering technology devices.
I wish each of you Great Success with Less Stress –
Oldcommguy™

Editor Profile Tim O’Neill is an independent technology consultant. He has over 40 years experience working in the WAN, Analog, ISDN, ATM and LAN test market. Tim has worked with companies like Navtel, Network General, Ganymede and ClearSight Networks and is now helping companies get technology verification and market validation. Tim is also the Chief Contributing Editor for LoveMyTool.com, a website designed to help network managers gain access to valuable information and real world solution stories. Tim is a patent holding, published and degreed engineer, who has seen this technology grow from Teletype (current loop) data analysis to today’s 10 Gigabit LAN’s focused on business applications with heavy compliance demands. Tim is heavily involved with helping Law Enforcement gain network and analysis knowledge and a regular speaker and trainer.
Tim can be reached at tim (at) oldcommguy (dot) com
**Please note – All Rights Reserved on the original material – Copyright 2010