AI, Communication Paradigm, and Tokenization

AI is increasingly using a complex model to analyze the complexity of the communication, structures in communication with none univariate level of semantics, trying to the outcome the best semantic in the adequate pragmatic. The way to recognize the syntactic shape could be extrapolated from the tokenization process until the complexity and the imbrication structure in the communication paradigm don't arise.

Tokenization is the process of segmenting running text into words and sentences.

Electronic text is a linear sequence of symbols (characters or words or phrases). Naturally, before any real text processing is to be done, the text needs to be segmented into linguistic units such as words, punctuation, numbers, alpha-numerics, etc. This process is called tokenization.

Tokenization is a kind of pre-processing in a sense; an identification of basic units to be processed. It is conventional to concentrate on pure analysis or generation while taking basic units for granted. Yet without these basic units clearly segregated it is impossible to carry out any analysis or generation.

The identification of units that do not need to be further decomposed for subsequent processing is an extremely important one. Errors made at this stage are very likely to induce more errors at later stages of text processing and are therefore very dangerous.

The errors arise principally like complexity as an element of constructivism in the descriptive process of the communication. This kind of mistakes are victims of too many simplifications inside the of NLP communication model paradigm.

Generally, if we have a low number of parameters or level involved, the systemic model is far to be trivial especially when the functional from the communication paradigm is partially or totally autocorrelated; as in the case of the self-referential structures.

From the perspective of complexity, the cybernetic argument is simple; connectivity between the components of a paradigm produces wholes with emergent properties that are different from those of these components. These emergent properties may be internal, as is the case with constructivism which allows the system to build to huge inductive challenges, or negative, as is the case for example with annihilation structure and human mistake. Connectivity among multiple components, as in the case of multiple paradigms is the trigger for non-linear dynamic systems and their hallmark is complexity.

The word complex start to be appropriated then when the number of parameter and level of autocorrelations between parameters, layers of paradigm involved and meanings extrapolates become relevant. Complexity is a way to define some systems in terms of relationships, flows and levels when the deterministic and/or analytics laws are not possibles to be revealed or even are completely absents as in the tokenization.

The systemic model is built using the concept of the goal and the meta-goal of a system. The pragmatic problem will be developed by the introduction of a concept: the "paradigm". The paradigm is the frame within which the meaning or the semantic will be developed or restructured, is used also differently but in linguistic-systemic we give this interpretation. Further, we will see how different frames of this kind when applied to a system model, such as the synchronic and diachronic frame, results in different perceptions a the same system. Different scales of time and functionality can influence strongly this perception.

The complexity perspective replaces the mono-level and multivariate complication with a descriptive element which is the paradigm model applied to another paradigm model - eventually is not the same semantic nature (E. Morin) -, for that, above defined, we associate the complexity concept to that one of the metaparadigm. The metalogic will be its characteristic design syntax.

The above model is a simplification of a paradigm model with functional and organica state.

Two main parts in the structures between complexity and metaparadigm:

• meta-structure

as in the above example.

• meta-goal

normally mentioned as latent goal or goal overlayer.

Types of meta-structures 

• functional meta-paradigm

referred to the functional part of the paradigm

• organic meta-paradigm

referred to the organic part of the paradigm

• abstract meta-paradigm

referred to the abstract construction over the level of the paradigm.

Construction of the model:

• Self-referential

it is like replied on itself.

• Structure

it means any paradigm model like NLP, Tokenization etc.

• Complexity 

overlayer (or layers) on the meaning (and/or meanings).

Using the correct structure of interpretation, AI, could take into account not only the optimal result in communication but the more appropriate, with this is underlined the difference between a simple reading of a text and his deep interpretation. Only in that way AI could travel along the line of the human complexity without only understanding at the first level of the meaning and avoiding the trivial simplifications.