CCSDS_study project

netzob-030/doc/documentation/source/language_specification/actor.rst

@@ -0,0 +1,23 @@
+.. _actor:
+
+Visiting a State Machine with an Actor
+======================================
+
+An **actor** (see :class:`~netzob.Simulator.Actor.Actor`) is a high-level representation of an entity that participates in a communication. An actor communicates with remote peers, in respect to an automaton (an actor is in fact a visitor of an automaton), and exchanges abstract representation of messages called Symbols.
+
+In the API, a visitor of a state machine is modeled using the Actor
+class.
+
+.. autoclass:: netzob.Simulator.Actor.Actor
+
+.. Note: we comment this section, as the figure is not referenced in the language specification.
+
+   .. figure:: ./img/transition_without_input_symbol_sequence.png
+      :align: center
+
+      Sequence diagram showing state transitions according to messages exchanged
+      between Alice and Bob.
+
+.. raw:: latex
+
+   \newpage

netzob-030/doc/documentation/source/language_specification/dataspec.rst

@@ -0,0 +1,506 @@
+
+.. _dataspec:
+
+.. _format_message_modeling:
+
+Format Message Modeling
+=======================
+
+The Netzob Description Language (ZDL) is the API exposed by the Netzob library
+to model data structures employed in communication protocols.
+This textual language has been designed in order to be easily understandable
+by a human. It enables the user to describe a protocol through dedicated
+*\*.zdl* files, which are independent of the API and core of the library.
+The ZDL language has been designed with attention to its expressiveness.
+In this chapter, firstly, the main concepts of the ZDL language are presented,
+then its expressiveness in terms of data types,
+constraints and relationships are explained.
+
+Format Message Modeling Concepts
+--------------------------------
+
+Definitions: Symbol, Field, Variable
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the Netzob library, the set of valid messages and their formats are
+represented through **symbols**. A symbol represents all the messages
+that share a similar objective from a protocol perspective.  For
+example, the HTTP_GET symbol would describe any HTTP request with the
+GET method being set. A symbol can be specialized into a context-valid
+message and a message can be abstracted into a symbol.
+
+A **field** describes a chunk of the symbol and is defined by a
+**definition domain**, representing the set of values the field handles.
+To support complex domains, a definition domain is represented by a tree where
+each vertex is a **Variable**. There are three kinds of variables:
+
+* **Data variables**, which describes data whose value is of a given **type**. Various types are provided with the library, such as String, Integer, Raw and BitArray.
+* **Relationship variables**, which make it possible to model a relationship between a variable and a list of variables or fields. Besides, relationships can be done between fields of different symbols, thus making it possible to model both **intra-symbol relationships** and **inter-symbol relationships**.
+* **Node variables**, which accept one or more children variables.
+
+Node variables can be used to construct complex definition domains,
+such as:
+
+* **Aggregate node variable**, which can be used to model a concatenation of
+  variables.
+* **Alternate node variable**, which can be used to model an alternative of
+  multiple variables.
+* **Repeat node variable**, which can be used to model a repetition of a
+  variable.
+* **Optional node variable**, which can be used to model a variable
+  that may or may not be present.
+
+As an illustration of these concepts, the following figure presents the
+definition of a Symbol structured with three Fields.
+The first field contains an alternative between String Data with a constant
+string and Integer Data with a constant value. The second field is String
+Data with a variable length string.
+The third field depicts an Integer whose value is the size of the second string.
+
+.. figure:: img/netzob_vocabulary_model.*
+   :align: center
+
+   Example of Symbol definition and relationships with Field and Variable objects.
+
+
+Abstraction and Specialization of Symbols
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The use of a symbolic model is required to represent the message formats of a protocol in a compact way. However, as the objective of this platform is to analyze the robustness of a target implementation, this implies that the testing tool should be able to exchange messages with this target. We therefore need to abstract received messages into symbols that can be used by the protocol model. Conversely, we also need to specialize symbols produced by the protocol model into valid messages. To achieve this, we use an **abstraction** method (*ABS*) and a **specialization** (*SPE*) method. As illustrated in the following figure, these methods play the role of an interface between the symbolic protocol model and a communication channel on which concrete messages transit.
+
+.. figure:: img/abstractionAndSpecialization.*
+   :align: center
+
+   Abstraction (ABS) and Specialization (SPE) methods are interfaces between the protocol symbols and the wire messages.
+
+To compute or verify the constraints and relationships that
+participate in the definition of the fields, the library relies on a
+:class:`~netzob.Model.Vocabulary.Domain.Variables.Memory.Memory`. This memory stores the value of previously captured or emitted
+fields. More precisely, the memory contains all the variables that are
+needed according to the field definition during the abstraction and
+specialization processes.
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Data Types
+-------------------
+
+The library enables the modeling of the following data types:
+
+* **Integer**: The Integer type is a wrapper for the Python integer object with the capability to express more constraints regarding the sign, endianness and unit size.
+* **HexaString**: The HexaString type makes it possible to describe a sequence of bytes of arbitrary size, with a hexastring notation (e.g. ``aabbcc``).
+* **BLOB / Raw**: The Raw type makes it possible to describe a sequence of bytes of arbitrary size, with a raw notation (e.g. ``\xaa\xbb\xcc``).
+* **String**: The String type makes it possible to describe a field that contains sequence of String characters.
+* **BitArray**: The BitArray type makes it possible to describe a field that contains a sequence of bits of arbitrary size.
+* **IPv4**: The IPv4 type makes it possible to encode a raw Python in an IPv4 representation, and conversely to decode an IPv4 representation into a raw object.
+* **Timestamp**: The Timestamp type makes it possible to define dates in a specific format (such as Windows, Unix or MacOS X formats).
+
+
+Data Types API
+^^^^^^^^^^^^^^
+
+Each data type provides the following API:
+
+.. autoclass:: netzob.Model.Vocabulary.Types.AbstractType.AbstractType()
+
+.. automethod:: netzob.Model.Vocabulary.Types.AbstractType.AbstractType.convert(typeClass)
+
+.. automethod:: netzob.Model.Vocabulary.Types.AbstractType.AbstractType.generate
+
+
+Some data types can have specific attributes regarding their endianness, sign and unit size. Values supported for those attributes are available through Python enumerations:
+
+.. autoclass:: netzob.Model.Vocabulary.Types.AbstractType.Endianness
+   :members:
+.. autoclass:: netzob.Model.Vocabulary.Types.AbstractType.Sign
+   :members:
+.. autoclass:: netzob.Model.Vocabulary.Types.AbstractType.UnitSize
+   :members:
+
+Data Types
+^^^^^^^^^^
+
+Supported data types are described in detail in this chapter.
+
+.. _integer_type:
+
+Integer Type
+++++++++++++
+
+In the API, the definition of an integer is done through the Integer class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.Integer.Integer
+
+BLOB / Raw Type
++++++++++++++++
+
+In the API, the definition of a BLOB type is made through the Raw class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.Raw.Raw(value=None, nbBytes=None, alphabet=None, default=None)
+
+HexaString Type
++++++++++++++++
+
+In the API, the definition of a hexastring type is made through the HexaString class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.HexaString.HexaString(value=None, nbBytes=None, default=None)
+
+String Type
++++++++++++
+
+In the API, the definition of an ASCII or Unicode type is made through the String class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.String.String(value=None, nbChars=None, encoding='utf-8', eos=[], default=None)
+
+BitArray Type
++++++++++++++
+
+In the API, the definition of a bitfield type is made through the BitArray class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.BitArray.BitArray(value=None, nbBits=None, default=None)
+
+IPv4 Type
++++++++++
+
+In the API, the definition of an IPv4 type is made through the IPv4 class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.IPv4.IPv4(value=None, network=None, endianness=Endianness.BIG, default=None)
+
+Timestamp Type
+++++++++++++++
+
+In the API, the definition of a timestamp type is done through the Timestamp class.
+
+.. autoclass:: netzob.Model.Vocabulary.Types.Timestamp.Timestamp(value=None, epoch=Epoch.UNIX, unity=Unity.SECOND, unitSize=UnitSize.SIZE_32, endianness=Endianness.BIG, sign=Sign.UNSIGNED, default=None)
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Fields
+---------------
+
+In the API, field modeling is done through the Field class.
+
+.. autoclass:: netzob.Model.Vocabulary.Field.Field
+   :members: specialize, abstract, getField, getSymbol, count
+
+   .. automethod:: netzob.Model.Vocabulary.Field.Field.copy()
+
+   .. automethod:: netzob.Model.Vocabulary.Field.Field.str_structure(preset=None)
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Variables
+------------------
+
+The definition domain of a field is represented by a tree of variables, containing leaf and node variables. Each variable follows a common API, which is described in the abstract class AbstractVariable:
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.AbstractVariable.AbstractVariable()
+   :members: count, isnode
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.AbstractVariable.AbstractVariable.copy()
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Data Variables
+-----------------------
+
+In the API, data variable modeling is made through the class Data.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Data.Data
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Data.Data.copy()
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Node Variables
+-----------------------
+
+Multiple variables can be combined to form a complex and precise
+specification of the values that are accepted by a field. Four complex
+variable types are provided:
+
+* **Aggregate node variables**, which can be used to model a concatenation of variables.
+* **Alternate node variables**, which can be used to model an alternative of multiple variables.
+* **Repeat node variables**, which can be used to model a repetition of a variable.
+* **Optional node variables**, which can be used to model a variable
+  that may or may not be present.
+
+Those node variables are described in detail in this chapter.
+
+Aggregate Domain
+^^^^^^^^^^^^^^^^
+
+In the API, the definition of a concatenation of variables is made through the Agg class.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Agg.Agg
+
+      .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Agg.Agg.copy()
+
+Alternate Domain
+^^^^^^^^^^^^^^^^
+
+In the API, the definition of an alternate of variables is made through the Alt class.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Alt.Alt
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Alt.Alt.copy()
+
+Repeat Domain
+^^^^^^^^^^^^^
+
+In the API, the definition of a repetition of variables, or sequence, is made through the Repeat class.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Repeat.Repeat
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Repeat.Repeat.copy()
+
+Optional Domain
+^^^^^^^^^^^^^^^
+
+In the API, the definition of a conditional variable is made through the Opt class.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Opt.Opt
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Nodes.Opt.Opt.copy()
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Fields with Relationship Variables
+-------------------------------------------
+
+The ZDL language defines constraints on variables, in order to handle relationships. Those constraints are leveraged during abstraction and specialization of messages. The API supports the following relationships.
+
+Value Relationships
+^^^^^^^^^^^^^^^^^^^
+
+In the API, the definition of a relationship with the value of another field is made through the Value class. This class enables the computation of the relationship result by a basic copy of the targeted field or by calling a callback function.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Value.Value
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Value.Value.copy()
+
+Size Relationships
+^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Size.Size
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Size.Size.copy()
+
+Padding Relationships
+^^^^^^^^^^^^^^^^^^^^^
+
+In the API, it is possible to model a structure with a padding through the Padding class.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Padding.Padding
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Padding.Padding.copy()
+
+Checksum Relationships
+^^^^^^^^^^^^^^^^^^^^^^
+
+The ZDL language enables the definition of checksum relationships between fields.
+
+**Checksum API**
+
+As an example, the API for the CRC16 checksum is as follows:
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC16.CRC16(targets)
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC16.CRC16.copy()
+
+**Available checksums**
+
+The following list shows the available checksums. The API for those checksums are similar to the CRC16 API.
+
+* :class:`CRC16(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC16.CRC16>`
+* :class:`CRC16DNP(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC16DNP.CRC16DNP>`
+* :class:`CRC16Kermit(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC16Kermit.CRC16Kermit>`
+* :class:`CRC16SICK(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC16SICK.CRC16SICK>`
+* :class:`CRC32(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRC32.CRC32>`
+* :class:`CRCCCITT(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.CRCCCITT.CRCCCITT>`
+* :class:`InternetChecksum(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Checksums.InternetChecksum.InternetChecksum>` (used in ICMP, UDP, IP, TCP protocols, as specified in :rfc:`1071`).
+
+Hash Relationships
+^^^^^^^^^^^^^^^^^^
+
+The ZDL language enables the definition of hash relationships between fields.
+
+**Hash API**
+
+As an example, the API for the MD5 hash is as follows:
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.MD5.MD5(targets)
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.MD5.MD5.copy()
+
+**Available hashes**
+
+The following list shows the available hashes. The API for those hashes are similar to the MD5 API.
+
+* :class:`MD5(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.MD5.MD5>`
+* :class:`SHA1(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.SHA1.SHA1>`
+* :class:`SHA1_96(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.SHA1_96.SHA1_96>`
+* :class:`SHA2_224(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.SHA2_224.SHA2_224>`
+* :class:`SHA2_256(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.SHA2_256.SHA2_256>`
+* :class:`SHA2_384(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.SHA2_384.SHA2_384>`
+* :class:`SHA2_512(targets) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hashes.SHA2_512.SHA2_512>`
+
+HMAC Relationships
+^^^^^^^^^^^^^^^^^^
+
+The ZDL language enables the definition of HMAC relationships between fields.
+
+**HMAC API**
+
+As an example, the API for the HMAC_MD5 is as follows:
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_MD5.HMAC_MD5(targets, key)
+
+   .. automethod:: netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_MD5.HMAC_MD5.copy()
+
+**Available HMACs**
+
+The following list shows the available HMACs. The API for those HMACs are similar to the HMAC_MD5 API.
+
+* :class:`HMAC_MD5(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_MD5.HMAC_MD5>`
+* :class:`HMAC_SHA1(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_SHA1.HMAC_SHA1>`
+* :class:`HMAC_SHA1_96(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_SHA1_96.HMAC_SHA1_96>`
+* :class:`HMAC_SHA2_224(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_SHA2_224.HMAC_SHA2_224>`
+* :class:`HMAC_SHA2_256(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_SHA2_256.HMAC_SHA2_256>`
+* :class:`HMAC_SHA2_384(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_SHA2_384.HMAC_SHA2_384>`
+* :class:`HMAC_SHA2_512(targets, key) <netzob.Model.Vocabulary.Domain.Variables.Leafs.Hmacs.HMAC_SHA2_512.HMAC_SHA2_512>`
+
+.. _modeling_symbols:
+  
+Modeling Symbols
+----------------
+
+In the API, symbol modeling is done through the Symbol class.
+
+.. autoclass:: netzob.Model.Vocabulary.Symbol.Symbol
+   :members: specialize, abstract, getField, count
+
+   .. automethod:: netzob.Model.Vocabulary.Symbol.Symbol.copy()
+
+   .. automethod:: netzob.Model.Vocabulary.Symbol.Symbol.str_structure(preset=None)
+
+.. raw:: latex
+
+   \newpage
+
+Configuring Symbol Content
+--------------------------
+
+Setting Field Values
+^^^^^^^^^^^^^^^^^^^^
+
+In the API, it is possible to control values that will be used in
+fields during symbol specialization. Such configuration can be done
+through the Preset class.
+
+.. autoclass:: netzob.Model.Vocabulary.Preset.Preset
+   :members: copy, update, bulk_set, clear
+
+Symbol with no Content
+^^^^^^^^^^^^^^^^^^^^^^
+
+A specific symbol may be used in the state machine to represent the
+absence of received symbol (EmptySymbol), when listening for incoming
+message, or the fact that nothing is going to be sent, when attempting
+to send something to the remote peer.
+
+.. autoclass:: netzob.Model.Vocabulary.EmptySymbol.EmptySymbol
+
+Relationships between Symbols and the Environment
+-------------------------------------------------
+
+In the API, a memory capability is provided in order to support
+relationships between variables, as well as variable persistence
+during the specialization and abstraction processes. This capability
+is described in the Memory class.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Memory.Memory
+   :members: memorize, hasValue, getValue, forget, copy
+
+In the API, the ability to specify relationships between successive
+messages or between messages and the environment is provided by the
+:class:`~netzob.Model.Vocabulary.Domain.Variables.Memory.Memory` class.
+
+
+**Relationships between fields of successive messages**
+
+The following example shows how to define a relationship between a
+received message and the next message to send. A memory is used to store the value of each variable. During the first call to :meth:`specialize` on the ``s1`` symbol, the value associated to the field ``f3`` is notably stored in memory, so that it can be retrieved when calling :meth:`specialize` on the ``s2`` symbol::
+
+  >>> from netzob.all import *
+  >>> f1 = Field(domain=String("hello"), name="F1")
+  >>> f2 = Field(domain=String(";"), name="F2")
+  >>> f3 = Field(domain=String(nbChars=(5,10)), name="F3")
+  >>> s1 = Symbol(fields=[f1, f2, f3], name="S1")
+  >>>
+  >>> f4 = Field(domain=String("master"), name="F4")
+  >>> f5 = Field(domain=String(">"), name="F5")
+  >>> f6 = Field(domain=Value(f3), name="F6")
+  >>> s2 = Symbol(fields=[f4, f5, f6])
+  >>>
+  >>> memory = Memory()
+  >>> m1 = next(s1.specialize(memory=memory))
+  >>> m2 = next(s2.specialize(memory=memory))
+  >>>
+  >>> m1[len("hello;"):] == m2[len("master>"):]
+  True
+
+
+**Relationships between a message field and the environment**
+
+The following example shows how to define a relationship between a
+message to send and an environment variable. The symbol is first
+defined, and then an environment variable is created. The first step
+consists in overloading the definition domain of the ``f9`` field to
+link the environment variable::
+
+  >>> from netzob.all import *
+  >>>
+  >>> # Symbol definition
+  >>> f7 = Field(domain=String("master"), name="F7")
+  >>> f8 = Field(domain=String(">"), name="F8")
+  >>> f9 = Field(domain=String(), name="F9")
+  >>> s3 = Symbol(fields=[f7, f8, f9])
+  >>>
+  >>> # Environment variables definition
+  >>> memory = Memory()
+  >>> env1 = Data(String(), name="env1")
+  >>> memory.memorize(env1, String("John").value)
+  >>>
+  >>> # Overloading f9 field definition to link the environment variable
+  >>> f9.domain = Value(env1)
+  >>>
+  >>> # Symbol specialization
+  >>> next(s3.specialize(memory=memory))
+  b'master>John'
+
+
+Persistence during Specialization and Abstraction of Symbols
+------------------------------------------------------------
+
+The values of variables defined in fields can have different assignment strategies, depending on their persistence and lifecycle.
+
+The Scope class provides a description of those strategies, along with some examples.
+
+.. autoclass:: netzob.Model.Vocabulary.Domain.Variables.Scope.Scope
+
+.. raw:: latex
+
+   \newpage

netzob-030/doc/documentation/source/language_specification/fuzzing.rst

@@ -0,0 +1,20 @@
+
+.. _fuzzing_symbols:
+
+Fuzzing Message Format
+----------------------
+
+The Preset class can be used to apply format message fuzzing. Fuzzing configuration is provided by the :meth:`fuzz` method.
+
+.. automethod:: netzob.Model.Vocabulary.Preset.Preset.fuzz(key, mode=None, generator=None, seed=None, counterMax=None, kwargs=None)
+
+.. automethod:: netzob.Model.Vocabulary.Preset.Preset.setFuzzingCounterMax
+
+.. automethod:: netzob.Model.Vocabulary.Preset.Preset.getFuzzingCounterMax
+
+.. automethod:: netzob.Model.Vocabulary.Preset.Preset.unset
+
+
+.. raw:: latex
+
+   \newpage

netzob-030/doc/documentation/source/language_specification/fuzzing_automata.rst

@@ -0,0 +1,12 @@
+.. _fuzzing_automata:
+
+Fuzzing Automata
+----------------
+
+Mutation of a protocol state machine is provided by the :meth:`mutate` method of the :class:`Automata <netzob.Model.Grammar.Automata.Automata>` class.
+
+.. automethod:: netzob.Model.Grammar.Automata.Automata.mutate
+
+.. raw:: latex
+
+   \newpage

netzob-030/doc/documentation/source/language_specification/protospec.rst

@@ -0,0 +1,17 @@
+
+.. _protospec:
+
+
+Protocol Modeling
+=================
+
+In the API, the Protocol class is the entry point for defining
+a complete protocol made of a state machine and different format
+messages.
+
+.. autoclass:: netzob.Model.Protocol.Protocol
+   :members: load_format
+
+.. raw:: latex
+
+   \newpage

netzob-030/doc/documentation/source/language_specification/statemachinespec.rst

@@ -0,0 +1,149 @@
+.. _statemachinespec:
+
+State Machine Modeling
+======================
+
+State Machine Modeling Concepts
+-------------------------------
+
+The ZDL language can be used to specify a **state machine**, or automaton, for a protocol. A state machine is based on two components: **States** and **Transitions**. A state represents the status of a service, and expects conditions to trigger the execution of a transition. A transition is a list of actions that will be executed when a condition is met at a specific state (such as the receipt of a network message). The list of actions may contain sending a network message, changing the value of session or global variables, moving to another state in the automaton, etc.
+
+The language defines three kinds of transition in an automaton:
+
+* **Standard transitions**: this represents a transition between two
+  states (an initial state and an end state) in an automaton. The
+  initial state and the end state can be the same.
+* **Opening channel transitions**: this represents a transition which, when
+  executed, requests to open the current underlying communication channel.
+* **Closing channel transitions**: this represents a transition which, when
+  executed, requests to close the current underlying communication channel.
+
+In the Netzob library, a state machine relies on symbols to trigger transitions between states. In order to represent the state machine structure, the library relies on a mathematical model based on a **Mealy machine** (cf. https://en.wikipedia.org/wiki/Mealy_machine). The library leverages this model by associating, for each transition, an input symbol and a list of output symbols, as shown on the figure below.
+
+.. figure:: img/state_machine.*
+   :align: center
+
+   Example of State Machine modeling with states, transitions, input and output symbols.
+
+Depending on the peer point of view, either an initiator (e.g. a client that starts a communication with a remote service) or a non initiator (e.g. a service that waits for input messages), the interpretation of the state machine is different. This intepretation is done with a state machine visitor that is called an :class:`~netzob.Simulator.Actor.Actor` in the API.
+
+From an **initiator point of view**, when the actor is at a specific state in the automaton, a random transition is taken amongst the available transitions. In the above example, two transitions, ``T1`` and ``T2``, are available at the state ``S1``. Then, the input symbol of the picked transition is specialized into a message and this message is emitted to the target. If the target replies, the actor abstracts the received message into a symbol, and checks if this symbol corresponds to one of the expected output symbols. If it matches, the transition succeeds and thus leads to the end state of the transition. In the above example, the transition ``T2`` would lead to the state ``S3``. If no response comes from the target, or if a wrong message is received, we leave the automaton.
+
+From a **non initiator point of view**, when at a specific state in the automaton, the actor waits for a network message. When one network message is received, it is abstracted into a symbol. Then, we retrieve the transition that has this symbol as input symbol. When a transition is retrieved, we randomly pick one symbol amongst the output symbols, and send this symbol to the remote peer. Finally, the transition leads to the end state of the transition.
+
+When the actor has to select a transition, or when the actor has to identify the current transition according to the received message, it is possible to influence this choice through the help of callback functions.
+
+Likewise, when the actor sends the input symbol, or when the actor sends an output symbol, it is possible to influence the selection of the symbol, through the help of callback functions or selection probability weight.
+
+In order to model hybrid state machines where a peer is able to send or receive symbols depending on the context, it is possible to change the initiator behavior at specific transitions. This is done through the :attr:`~netzob.Model.Grammar.Transitions.Transition.Transition.inverseInitiator` attribute on :class:`~netzob.Model.Grammar.Transitions.Transition.Transition` objects. When setting this attribute to ``True`` on a transition, an actor will inverse the way symbols are exchanged (e.g. an initiator actor will first wait for an input symbol and then send one of the output symbols).
+
+A **Memory** (see :class:`~netzob.Model.Vocabulary.Domain.Variables.Memory.Memory`) is used to keep track of a context for a specific communication. This memory can leverage variable from the protocol or even the environment. The memory is initialized at the beginning of the communication, and its internal state evolves throughout the exchanged messages.
+
+
+.. Besides, two extensions allow refining the state machine model:
+
+.. * The capability to define a reaction time on a transition. This reaction time between receiving a specific symbol and sending the output symbol will be enforced by the library.
+.. * The capability to provide indeterminism on output symbols. The library enables the user to model a transition which, for a sequence of input symbols, associates many sequences of output symbols. The chosen sequence of output symbol is selected randomly.
+
+.. raw:: latex
+
+   \newpage
+
+
+Modeling States
+---------------
+
+In the API, automaton states are modeled through the State class.
+
+.. autoclass:: netzob.Model.Grammar.States.State.State
+   :members: copy
+
+.. raw:: latex
+
+   \newpage
+
+Modeling Transitions
+--------------------
+
+The available transitions are detailed in this chapter.
+
+.. autoclass:: netzob.Model.Grammar.Transitions.Transition.Transition(startState, endState, inputSymbol=None, outputSymbols=None, name=None)
+   :members: copy
+
+.. autoclass:: netzob.Model.Grammar.Transitions.OpenChannelTransition.OpenChannelTransition(startState, endState, name=None)
+   :members: copy
+
+.. autoclass:: netzob.Model.Grammar.Transitions.CloseChannelTransition.CloseChannelTransition(startState, endState, name=None)
+   :members: copy
+
+.. raw:: latex
+
+   \newpage
+
+
+Taking Control over Emitted Symbol and Selected Transition
+----------------------------------------------------------
+
+A state may have different available transitions to other states. It
+is possible to filter those available transitions in order to limit
+them or to force a specific transition to be taken. The filtering
+capability works by adding callbacks through the
+:meth:`add_cbk_filter_transitions` method on a
+:class:`~netzob.Model.Grammar.States.State.State` instance.
+
+
+.. automethod:: netzob.Model.Grammar.States.State.State.add_cbk_filter_transitions
+
+When a transition is selected, it is possible to modify it by adding
+callbacks through the :meth:`add_cbk_modify_transition` method on a
+:class:`~netzob.Model.Grammar.States.State.State` instance.
+
+
+.. automethod:: netzob.Model.Grammar.States.State.State.add_cbk_modify_transition
+
+Besides, during execution of a transition, it is possible to change
+the symbol that will be sent to the remote peer, by adding callbacks
+through the :meth:`add_cbk_modify_symbol` method on a
+:class:`~netzob.Model.Grammar.Transitions.Transition.Transition` instance.
+
+.. automethod:: netzob.Model.Grammar.Transitions.Transition.Transition.add_cbk_modify_symbol
+
+
+Executing Actions during Transitions
+------------------------------------
+
+It is possible to execute specific actions during transitions, after sending or receiving a symbol, by adding
+callbacks through the :meth:`add_cbk_action` method on a
+:class:`~netzob.Model.Grammar.Transitions.Transition.Transition` instance. The typical usage of this callback is that it is possible to manipulate the memory context of the automaton after sending or receiving a symbol.
+
+When specifying such callback on a transition, this callback is then called twice for a transition: in an initiator context, the callback is first called after sending the input symbol, and then called after receiving one of the output symbols; while in a non initiator context, the callback is called after receiving the input symbol, and then called after sending one of the output symbols.
+
+.. automethod:: netzob.Model.Grammar.Transitions.Transition.Transition.add_cbk_action
+
+
+Summary of States and Transitions Processing
+--------------------------------------------
+
+The following figure gives a summary of the sequence of operations during states and transitions processing.
+
+.. figure:: img/Grammar_procedure.*
+   :align: center
+   :scale: 70 %
+
+   Sequence of operations during states and transitions processing
+
+
+Modeling Automata
+-----------------
+
+In the API, an automaton is made of a list of permitted symbols
+and an initial state. An automaton is modeled using the Automata class.
+
+.. autoclass:: netzob.Model.Grammar.Automata.Automata
+   :members: getStates, getState, getTransitions, getTransition, set_cbk_read_symbol_timeout,
+             set_cbk_read_unexpected_symbol, set_cbk_read_unknown_symbol,
+             generateDotCode, generateASCII, copy
+
+.. raw:: latex
+
+   \newpage

netzob-030/doc/documentation/source/language_specification/trafficgeneration.rst

@@ -0,0 +1,150 @@
+.. _trafficgeneration:
+
+Sending and Receiving Messages
+==============================
+
+Underlying Concepts
+-------------------
+
+In the Netzob library, a **communication channel** is an element allowing a connection to a remote device. Generally, if the device is connected with an Ethernet network, the channel includes a socket object and all the properties used to configure it. The channel also provides the connection status and send/receive APIs.
+
+Some specific channels make it possible to access and manipulate the underlying protocol header. These channels are prefixed with the term ``Custom``. The underlying protocol header takes the form of a :class:`Symbol <netzob.Model.Vocabulary.Symbol.Symbol>` for which we can specify a :class:`Preset <netzob.Model.Vocabulary.Preset.Preset>` configuration.
+
+These elements are described in this chapter.
+
+
+.. _trafficgeneration_channel_list:
+
+Communication Channel API
+-------------------------
+
+Each communication channel provides the following API:
+
+.. autoclass:: netzob.Simulator.AbstractChannel.AbstractChannel()
+   :members: open, close, __enter__, __exit__, read, write, write_map, flush, sendReceive, setSendLimit, clearSendLimit, set_rate, unset_rate
+
+.. note::
+   There are two ways to open and close a channel.
+   **Both methods provide the same behavior**.
+
+   1. either by using the related methods:
+      :meth:`~netzob.Simulator.AbstractChannel.AbstractChannel.open` and
+      :meth:`~netzob.Simulator.AbstractChannel.AbstractChannel.close`.
+      Example:
+
+      .. code-block:: python
+
+         channel.open()
+         try:
+             channel.write(b'abcd')
+         finally:
+             channel.close()
+
+   2. or by using Python contexts capability provided by the ``with`` statement
+      and following methods:
+      :meth:`~netzob.Simulator.AbstractChannel.AbstractChannel.__enter__` and
+      :meth:`~netzob.Simulator.AbstractChannel.AbstractChannel.__exit__`.
+      Example:
+
+      .. code-block:: python
+
+         with channel:
+             channel.write(b'abcd')
+
+Builder classes (see `Build pattern <https://en.wikipedia.org/wiki/Builder_pattern>`_)
+are also available for each communication channel. They could be used to create
+an instance of the channel class using generic keys.
+
+This API is available through the following class:
+
+.. autoclass:: netzob.Simulator.ChannelBuilder.ChannelBuilder
+   :members: set, set_map, build
+
+.. _channels:
+                
+Available Communication Channels
+--------------------------------
+
+The available communication channels are as follows:
+
+* :class:`~netzob.Simulator.Channels.RawEthernetChannel.RawEthernetChannel`: this channel sends/receives Raw Ethernet frames.
+* :class:`~netzob.Simulator.Channels.CustomEthernetChannel.CustomEthernetChannel`: this channel sends/receives Ethernet frames (with Ethernet header computed by this channel).
+* :class:`~netzob.Simulator.Channels.CustomIPChannel.CustomIPChannel`: this channel sends/receives IP payloads (with IP header computed by this channel). 
+* :class:`~netzob.Simulator.Channels.IPChannel.IPChannel`: this channel sends/receives IP payloads (with IP header computed by the OS kernel).
+* :class:`~netzob.Simulator.Channels.UDPClient.UDPClient`: this channel provides the connection of a client to a specific IP:Port server over a UDP socket.
+* :class:`~netzob.Simulator.Channels.TCPClient.TCPClient`: this channel provides the connection of a client to a specific IP:Port server over a TCP socket.
+* :class:`~netzob.Simulator.Channels.UDPServer.UDPServer`: this channel provides a server listening to a specific IP:Port over a UDP socket.
+* :class:`~netzob.Simulator.Channels.TCPServer.TCPServer`: this channel provides a server listening to a specific IP:Port over a TCP socket.
+* :class:`~netzob.Simulator.Channels.SSLClient.SSLClient`: this channel provides the connection of a client to a specific IP:Port server over a TCP/SSL socket.
+* :class:`~netzob.Simulator.Channels.DebugChannel.DebugChannel`: this channel provides a way to log I/Os into a specific stream.
+
+.. _trafficgeneration_channels:
+     
+Each communication channel, with their associated builder class, is described
+in the following sub-chapters.
+
+RawEthernetChannel channel
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.RawEthernetChannel.RawEthernetChannel
+.. autoclass:: netzob.Simulator.Channels.RawEthernetChannel.RawEthernetChannelBuilder
+
+CustomEthernetChannel channel
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.CustomEthernetChannel.CustomEthernetChannel
+   :members: setProtocol
+.. autoclass:: netzob.Simulator.Channels.CustomEthernetChannel.CustomEthernetChannelBuilder
+
+CustomIPChannel channel
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.CustomIPChannel.CustomIPChannel
+.. autoclass:: netzob.Simulator.Channels.CustomIPChannel.CustomIPChannelBuilder
+
+IPChannel channel
+^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.IPChannel.IPChannel
+.. autoclass:: netzob.Simulator.Channels.IPChannel.IPChannelBuilder
+
+UDPClient channel
+^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.UDPClient.UDPClient
+.. autoclass:: netzob.Simulator.Channels.UDPClient.UDPClientBuilder
+
+TCPClient channel
+^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.TCPClient.TCPClient
+.. autoclass:: netzob.Simulator.Channels.TCPClient.TCPClientBuilder
+
+UDPServer channel
+^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.UDPServer.UDPServer
+.. autoclass:: netzob.Simulator.Channels.UDPServer.UDPServerBuilder
+
+TCPServer channel
+^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.TCPServer.TCPServer
+.. autoclass:: netzob.Simulator.Channels.TCPServer.TCPServerBuilder
+
+SSLClient channel
+^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.SSLClient.SSLClient
+.. autoclass:: netzob.Simulator.Channels.SSLClient.SSLClientBuilder
+
+DebugChannel channel
+^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: netzob.Simulator.Channels.DebugChannel.DebugChannel
+.. autoclass:: netzob.Simulator.Channels.DebugChannel.DebugChannelBuilder
+
+
+.. raw:: latex
+
+   \newpage