The "UNIAN-Monitor" software complex with the help of customizable filters or categorization in 'Latest News' section allows users to choose that volume of publications, which coincides with their professional tasks. In other words the system aggregates publications from hundreds of sources and with the help of several customizable filters the useless information.
General scheme of Web-Observer is shown on the picture below:
Technically this can be described in the following manner:
"The programmable generator of events sets the frequency of actions, performed by the system: screening of sources, generation of reports, etc. After screening the sources and detection of new materials, the procedure of passing the page/pages to parsing module is launched. The parser, using the predefined style of the source, extracts the necessary data from HTML structure and stores them to the database"
The system saves only the text of publication and its attributes, but not the whole page. It substantially reduces the sizes of the stored information and consequently simplifies and speeds up search in the database of publications. When you need to review the document directly on the source, you may follow the link included in the end of each article, which shows the source page the article was retrieved from.
After checking the article, the system defines whether it coincides with search criteria and forms the custom articles feed, which is then delivered to client's e-mail or mobile phone via SMS.