Data Privacy Day

Today is Data Privacy Day. This occurs every year on the 28 January. Data Privacy Day is an annual international celebration designed to promote awareness about privacy and education about best privacy practices.

Click on the link above to go to the Data Privacy page for more information and a list of resources to aid you in keeping your data safe while online.

To some extent, this is true. For instance, JAVA can be considered a closed world, as it has it’s own tool set and virtual machine. Vivek uses Unix as the example of a open world, a loosely coupled system of programs and utilities which can be split apart and run on multiple systems; Unix being only one of many.

The multivalued database systems definitely fall into the closed world category. Considering that the heritage of all mvdbms comes from PICK, which was a standalone operating system with a database and programming tools added. Is this an issue? Vivek considers that all closed worlds eventually die; the multivalued market is not large, but also not dead yet. But then, it is so closed, that only VAR’s can get full access to development tools anyway. Maybe this is how it has survived for so long.

On another note, the Programming as Theory-Building sounds very interesting.

While browsing my old Universities library last year, I discovered an old text book on the subject of Object-Oriented Database Management by Kemper & Moerkotte. What is of real interest is that “Chapter 6 – Extensions of Relational DBMSs” has a rather good coverage of the “6.6 – The Nested Relational Model NF2″ with a bunch of references or citations to papers all written around about 1986.

So, what happen within the database academic world in the mid 1980′s that NF² was all the rage?

Now, the text book is analyzing the NF² model as an extension to the relational SQL database model and comparing all relational database models to an object-oriented model. So, as object-oriented anything is all so much better than anything else (good-bye and farewell Forth), it is a bit biased.

The mathematics for the set-theory around the “6.6.2 – Formal Definition of NF² Relations” section is good; for those of you whom like maths. The theory eventually gives way to a section on the query language to support NF²; all of which is in a modified/extended form of SQL. That is OK, but it is easy to forget that the multi-value data model that is implemented within the PICK model is  single file based; a data selection can only be performed or centered around one table or file at a time, with the use of translations to other files is within virtual fields. SQL has the ability to dynamically link tables together within the selection process; hence the relational side of a RDBMS. It’s really all good stuff…

The summary of the chapter leads to the conclusion that the NF² data model is “…really a hybrid of the relational and the hierarchical data model.” and there are problems with data redundancy, ownership of objects, and sharing of objects due to the nesting. The NF² model provides a extremely concise data structure, but the side-effect is that the extended SQL query becomes non-trivial and just as complex as for a pure relational model schema. OK!

The annotated bibliography at the end of the chapter lists an interesting selection of papers and presentations regarding NF² data structuring concepts. They are all based approximately around 1986 to 1987.  There are references to the AIM-P and DASDBS database prototypes, and the HDBL language. And then there is R²D² – a relational database system; not the inverted-rubbish-bin-on-wheels robot. There is NO references to the PICK data model. Rather sad.

So, I am currently attempting to beg, borrow, and steal the text of these references and citations, in an attempt to follow through and see that all the excitement is all about. Why? Well, this is the first time that I have come across some form of academic analysis of the NF² model. There is nothing contemporary  happening on the subject – shame on the multi-value database vendors (remember, a white-paper is a SALES tool – like a brochure; not an unbiased scientific/academic paper).

So, if you have any information regarding the above NF² systems, leave a comment please.

And I will be writing more when I  found out more. It would be nice to be able to sit across the boardroom table and throw out some academic references on the multi-value model when the selling gets tough.

All good stuff really…

At the time, I thought – what the … But then again, there just might be something in this:

As I have stated before, I consider that the U2/PICK database model to be very ER (Entity-Relationship) Model based; hence the name I use: DDERDBMS – Dictionary-Driven Entity-Relationship DataBase Management System.

This makes sense, as the data is structured as ER relationships, and in most cases does not need to be normalized (think back to your text book ERD chapter). The actual fields, attributes, and values are then defined at the dictionary level.; where the dictionary definitions actually ‘point’ to fixed positions within the database e.g. <1,2>.

But, what about the UniBasic coding environment? Fields are not actually referenced by the data dictionary definitions. They are referenced as either a static multidimensional array or as a dynamic array,  essentially as a document structure. Of cause, like many other people, I have written or used some sort of utility to create a series of EQUATE statements for each data file, which can be INCLUDE’d within the appropriate UniBasic source code so some sort of relevance between the field and value numbers and names exist. Even SB+ has such a utility. But, even then, the ASSOCIATIVE relationship between fields and values are not known within UniBasic. It’s all manually managed outside of the programming environment (think paper, the brain,and a good memory).

But, at the end of the day, all references of fields and values are via positional numbers, not by actual field names. There are significant advantages and disadvantages, as expected; lack of self-documenting code is the first one that comes to mind as a major disadvantage.

So, what is the big deal? Well, over the last couple of months, longer maybe, there has been spurts of chatter within the couple of U2 forums about the future of U2 (market is growing?) and job opportunities etc. Ignoring all the politics which exists between the Haves (VAR’s) and the Have-Nots (employees and contractors) the general consensus is that the market has changed. This is  due to the way that U2 is used. It is no longer used as a complete front-end to back-end development environment. There are now multiple front-ends to consider (desktop, phone, web, tablet!) with better tools to create and maintain them. U2 is being relegated to the back-end.

This is a trend that no-one should be surprised with, as in the distant past, PICK -type databases were the OS, database, programming environment, and the client interface. Over time each of these functions have been replaced with more generic, and better alternatives e.g. the OS is now either MS-Windows or Linux (Unix).

So, if you are using a client/server structure for access to the U2 database, then there is probability an API. Therefore, using U2 Access for report generation must be on the decline, and the use of the dictionary definitions for a data file are for documentation reasons only? Well, using the field definitions does come in handy for implementing indexing and the odd SELECT statement, I suppose.

Which comes back to the UniBasic programming paradigm environment. Reading a record returns a collection of  multiple collections of multiple values, which all have positional relevance. Something like a document-oriented database.

This also leads to what a NoSQL database actually is – and is U2 a NoSQL database?

I personally believe that U2 and PICK type databases , in general, are not NoSQL.

They are DDERDBMS – Dictionary Driven Entity Relationship DataBase Management Systems.

Just rolls off the tongue…

Have a look at – http://www.red-gate.com/products/sql-development/sql-source-control/entrypage/assets/RG_Guide_to_SQL_Server_Dev.pdf

Also Knuth has a sense of humor which lightens the reading a wee bit, so I am glad to find an article which lists one-liner jokes about Knuth – http://jjinux.blogspot.com/2011/03/humor-donald-knuth-jokes.html

Guess what? It still exists in 10.3.7.0 on MS-Windows. So, here is how to time travel with @DATE.

Before we start, take this poll:

[poll id="3"]

I had found a problem with the SUBR call in a I-Type field definition. After testing, it appears to be within a BASIC program as well and is to do with the pass-by-value and pass-by-reference differences. So I thought.

The routine is passed a date, which I use @DATE as the default for this month. For other months or days, I would add or subtract 30,60,90 etc.

SUBR("CALC.DTE.SUBR",MODE,@DATE)

Within the routine, the date variable is changed – adding 1 or 2 days based on some simple calculations. What I found was that the system date was changing as well!

So, I have below two routines to test the theory:

PROGRAM RUN.TEST.DATE

RESULT = 0

PRINT "S@DATE = ", @DATE, OCONV(@DATE, "D4")
RESULT = SUBR("TEST.DATE", @DATE)
PRINT "E@DATE = ", @DATE, OCONV(@DATE, "D4")

END

and

SUBROUTINE TEST.DATE(RESULT, PARAM1 )

PRINT "PDATE = ", PARAM1, OCONV(PARAM1, "D4")
PRINT "@DATE = ", @DATE, OCONV(@DATE, "D4")
PRINT "DATE() = ", DATE(), OCONV(DATE(), "D4")

PARAM1 += 1
PRINT "ADDED A DAY"

PRINT "PDATE = ", PARAM1, OCONV(PARAM1, "D4")
PRINT "@DATE = ", @DATE, OCONV(@DATE, "D4")
PRINT "DATE() = ", DATE(), OCONV(DATE(), "D4")

RESULT = 0

RETURN
END

The output will be:

S@DATE =  15765     28 FEB 2011
PDATE =   15765     28 FEB 2011
@DATE =   15765     28 FEB 2011
DATE() =  15765     28 FEB 2011
ADDED A DAY
PDATE =   15766     01 MAR 2011
@DATE =   15766     01 MAR 2011
DATE() =  15765     28 FEB 2011
E@DATE =  15766     01 MAR 2011

So, if you follow the PRINT statements, I start in the PROGRAM with 28 Feb, pass that through with @DATE as the parameter (pass by reference) and then added one day to it. The parameter within the subroutine changes to 1st March (PDATE) and also the date @DATE has also changed!

Since the parameter was passed by reference, the date (via @DATE) has also changed within the calling program (E@DATE).

If I change the passing of the parameter to pass-by-value by placing the parameter within brackets, the problem goes away.

If I change the passing parameter to DATE(), then the problem does not occur. So, what is the difference between @DATE and DATE()?

Well, I did discover something which I was not aware of: @DATE is the “internal date when the program was invoked”. This is from the UniBasic Reference documentation. I had always assumed that it was the system date now. Appears not! You learn something new everyday…

And DATE() “… return(s) the numeric value of the internal system date.” That makes sense. OK…

So, @DATE (and assuming that all derived @ values e.g. @DAY) are actually system variables that are created at the time of the program is invoked (run). And they are mutable. That is, they can be changed.

And this is all correct!

Oh, Yes. In the Appendix E of the UniVerse Basic Manual, there is a list of the @variables and “…denoted by an asterisk (*) are read-only. All others can be changed by the user.”

@DATE, @DAY, @MONTH and a whole bunch of others are mutable.

So, if you want to fudge your time sheets within your departments time-sheet software, in the next maintenance cycle, refactor to use @DATE and @TIME. Only joking, of cause.

So, recapping; DATE() is the CURRENT system date; @DATE WAS the date that the program was started, and if you started it late, you can change it. And all the subroutine parameter passing? Not really relevant; it was how I re-found the knowledge of @DATE.

Now, who can be honest here? How many readers thought that @DATE and DATE() were interchangeable?

And now?

[poll id="4"]

As most IT people have found, I also repeatedly find MS-Excel used for all sorts of patchwork business processes. Some have even been clever. But the MS-Excel silo issue always raises it ugly head.

But the relationship between these two methods and in particular between SUBROUTINE and FUNCTION is rather interesting.

Internal Subroutine

The internal subroutine GOSUB is really just a hacked GOTO with a automatically stored jump back or RETURN.

The mechanism of use is to call the subroutine via the GOSUB statement with a label which refers to the code point to transfer to. Ending and exiting the subroutine is via the RETURN statement.

PROGRAM XYZ
...
GOSUB READSTUFF:
...
STOP
READSTUFF: *Routine to Read stuff
...
RETURN

The internal subroutine has no local scope capabilities and if not carefully written, variable naming can become an issue as they are global within the complete routine.

But the GOSUB can very handy when the processing and code logic becomes large and repeatable. I also think that the ON GOSUB is rather neat. I have ignored the RETURN TO statement, as like a GOTO, it is just plain evil.

The external subroutine on the other hand consists within it’s own right as a compiled and cataloged routine and comes within two forms – SUBROUTINE and FUNCTION.

External Subroutine

The subroutine is a separate routine which has been compiled and cataloged and that can be called by any other routine by the CALL statement. The routine must start with a SUBROUTINE statement which provides a name and a list of parameters that will be passed:

SUBROUTINE TEST(VAR1, VAR2)
...
RETURN

and therefore,

PROGRAM XYZ
...
CALL TEST(TEST1, TEST2)
...
END

Strangely enough, how the variables are passed is governed by the calling process, not the SUBROUTINE definition. By default the parameters are passed by reference; that is, the pointers to the parameter variables are passed and the values are not copied. Hence, within the SUBROUTINE, any changes to the passed variables will be reflected within the calling routine. This method is used as it is faster at transferring variables than by copying a value, where each parameter must be copied to a new variable,  especially if a large number of parameters are to be passed or if the parameters are arrays.

There are two common methods of dealing with this situation: define the passed parameters as pass-by values (enclose within brackets) or copy the parameters to new variables within the subroutine:

PROGRAM XYZ
...
CALL TEST((TEST1), (TEST2))
...
END

or

SUBROUTINE TEST(VARTEMP1, VARTEMP2)
VAR1 = VARTEMP1
VAR2 = VARTEMP2
...
RETURN

With the SUBROUTINE, any return values are transferred back via one of the calling parameters. So, I prefer to use method two, copying around the input values as it is not really appropriate to have this controlled by the calling routine.

PROGRAM XYZ
...
CALL TEST(INVAR1, INVAR2, RESULT)
...
END

and

SUBROUTINE TEST(INVARTEMP1, INVARTEMP2, RESULT)
INVAR1 = INVARTEMP1
INVAR2 = INVARTEMP2
...
RETURN

External Function

The FUNCTION is similar to the SUBROUTINE except that a value is returned by the RETURN statement and the calling process is similar to a normal function with UniBasic.

The routine must be defined as a function by the FUNCTION statement on the first line of the routine and within the calling routine, the function name must be defined as a function via the DEFFUN statement:

FUNCTION TEST(VAR1, VAR2)
...
RETURN(RESULT)

and

PROGRAM TESTFUN
DEFFUN TEST(VAR1, VAR2)
...
RESULT = TEST(VAR1, VAR2)
...
END

The parameters within the DEFFUN statement are dummies, and are only used to define what type of data is being transferred (string or array, for example). The interesting part is that there is an ‘invisible’ variable created within the function parameter list for the return value. It is placed as the first variable within the argument list. This is equivalent to:

FUNCTION TEST(RESULT, VAR1, VAR2)

SUBR

The use of the SUBR statement is similar to the FUNCTION except the name of the function to call is passed as the first parameter. Also, the returned parameter must be the second parameter within the list.

There is a lengthy description within the U2documentation – the UniBasic Reference Manual – about how to structure a SUBROUTINE to work with the SUBR.

So, can a function and a subroutine be interchangeable when used with a SUBR? Yes, if the subroutine follows the return-as-first-parameter rule.

For example, here are a subroutine and function:

SUBROUTINE TESTSUB(RESULT, PARAM1, PARAM2)
  RESULT = "TESTSUB = " : PARAM1 + PARAM2
RETURN

and

FUNCTION TESTFUNC(PARAM1, PARAM2)
  RESULT = "TESTFUNC = " : PARAM1 * PARAM2
RETURN(RESULT)

and

PROGRAM TEST
  EQUATE EMPTY.STR TO ""

  PRINT SUBR("TESTSUB", 2, 3)
  PRINT SUBR("TESTFUNC", 2, 3)

  DEFFUN TESTSUB(PARAM1, PARAM2)
  PRINT "DEFFUN = " : TESTSUB(1,2)

  RESULT = EMPTY.STR
  CALL TESTFUNC(RESULT, 3, 4)
  PRINT "FUNC AS SUB = " : RESULT
END

and the output will be:

TESTSUB = 5
TESTFUNC = 6
DEFFUN = TESTSUB = 3
FUNC AS SUB = TESTFUNC = 12

The SUBROUTINE and FUNCTION are interchangeable as long as the parameters adhere to the structure of the result variable as the first variable in the SUBROUTINE list.

As the SUBR is normally used within a derived field, it is possible to write one routine that can be used within a UniBasic program and within a dictionary field (derived) definition without any changes. This is an important concept that is often overlooked within U2.

And yes, if you had noticed, everything becomes a function! Is this so bad?

This can also be extended into SB+ by the use of a small PARAGRAPH to wrap the CALL statement. And as I have mentioned before, SB+ is a highly functional paradigm.

So, back to DRY – Don’t Repeat Yourself – use one routine for each of your business logic and processes, in all places within U2.