Do LLMs Beat Classic Models at Catching Network Attacks?
Small language models learned to flag network attacks and scored above 95 percent. But a plain classic method, XGBoost, still won, faster and far cheaper.
In one paragraph
Researchers at the University of Zaragoza tested whether small language models can spot attacks in smart-home network traffic better than classic machine learning. Fine-tuned language models scored above 95 percent on the F1 measure and beat every deep learning model, but XGBoost, a classic tree-based method, still came out on top at 0.9696 while using about 4% of the processor. The language models were also about 10,000 times slower. The evidence is limited: one dataset, small models, and a single run per model.
How did the researchers test language models on network attacks?
The authors used a public dataset, CIC IoT 2023, recorded from 105 real smart devices such as cameras, sensors and smart speakers. They kept four kinds of traffic: ordinary household traffic and three attack types, namely flooding attacks (DDoS), the Mirai botnet, which hijacks smart gadgets, and scanning, where an attacker probes a network for weak spots.
Each connection was summarised as a single record, called a flow, with facts such as its duration, bytes sent each way and how it ended. Timestamps, IP addresses and ports were removed, so no model could simply memorise where attacks came from.
Nine models then faced the same data: three classic machine learning methods (including XGBoost), three deep learning networks and three small language models (GPT-2, GPT-Neo-125M and LLaMA-3.2-1B). The language models read each flow as a line of text and were mostly fine-tuned, meaning a general model gets brief extra training on one job, on a single graphics card with 6 GB of memory. Every model trained on 10,000 or 50,000 flows per class and was tested on unseen flows.
Which model was best at detecting attacks?
XGBoost had the best scores overall. Its F1-score, a measure that rewards catching attacks while avoiding false alarms, reached 0.9696 with the larger training set, and it used about 4% of the processor during testing.
The language models did well too. Their scores stayed above 95 percent, and the best one, GPT-Neo after fine-tuning, reached 0.9618 on the four-way task. That beat every deep learning model; two of them, GRU and LeNet-5, fell below 0.85 on the same task, mostly by confusing ordinary traffic with scanning.
The big difference was speed. The fastest language model setup handled at most 517 flows per second, against 1,581,027 for XGBoost, making the language models about 10,000 times slower. Their mistakes were also slightly more frequent: in the attack-or-not test, XGBoost let 3.3% of attack traffic pass as normal, against 3.8% for the best language model.
Should security tools use language models for intrusion detection?
Intrusion detection software often runs close to the devices it protects, such as a small box on a home or office network, where memory and processing power are limited. In this study the classic models needed under a megabyte of disk space and no graphics card, while the language models needed gigabytes of storage and a graphics card.
A fair reading is that, for sorting large volumes of structured traffic, the classic method remains the one to beat: faster, cheaper to run and slightly more accurate. The language models still earned a place in the conversation, since they came close on accuracy and held steady even with the smaller training set.
The authors suggest a combined design for future work, in which a fast classic model sorts traffic and a language model is brought in for the hard or ambiguous cases, possibly to describe an alert in plain language.
What are the limits of this intrusion detection study?
Read before you quote it
Several limits shape how far the results reach:
- Small models only. The largest language model had about 1.3 billion parameters; larger models such as GPT-4 or LLaMA-2-7B were not tested and may do better.
- Single runs. Each model was trained and scored once, without repeated runs or cross-validation, so the small gaps between models carry no statistical test.
- One dataset, known attacks. All results come from one dataset of known attack types, so new kinds of attack or other networks could change the picture.
- Opaque decisions. The explanation method used (SHAP) only worked on the tree models, so how the language models reached their decisions is unknown.
The study also does not show that language models are useless for security, since they came within about a point of the winner. It does not show that a language model can explain its alerts in plain language; that was left for future work.
- It does not show that language models are useless for security: they came within about a point of XGBoost.
- It does not show how large, modern chat models like GPT-4 would perform on this task.
- It does not show that language models can explain their alerts in plain language; that was left for future work.
- Only three small language models were tested; larger models such as GPT-4 or LLaMA-2-7B may do better.
- Each model was trained and scored once, with no repeated runs or cross-validation, so the small gaps between models carry no statistical test.
- All results come from one dataset with known attack types, so they may not hold for new kinds of attack or other networks.
- The explanation tools (SHAP) could only be applied to the tree models, so how the language models reached their decisions is unknown.
Takeaways
- saysSmall language models scored above ninety-five percent at spotting attacks, but XGBoost still won at almost ninety-seven.
- saysThe language models were about ten thousand times slower than the classic methods.
- saysThis was one dataset with known attacks and small models, so bigger models and new kinds of attack are still untested.
- explainsRemoving timestamps, IP addresses and ports keeps models from memorising where attacks come from instead of learning how they behave.
- interpretsA fair reading is that a classic model should stay the default for high-volume traffic, with language models reserved for harder cases.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Are language models better than XGBoost at detecting network attacks?
Not in this study: XGBoost scored slightly higher, and ran about 10,000 times faster. Hear the full answer in the app →
Why did the researchers remove IP addresses from the data?
So the models had to learn how attacks behave instead of memorising where they came from. Ask in the app →
Which traffic was hardest to classify?
Scanning, which was often confused with ordinary household traffic by both the best language model and XGBoost. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Paper licence: CC BY 4.0. Corrections: support@papersays.com.