Øyvind Teig

036 - WYSIWYG semantics

2011-12-27T14:35:00.024+01:00

WYSIWYG of course means What You See is What You Get. The semantics describes what happens. So, when you look at the source code in this process (thread, task), are you certain that you don't need to know anything (other than the message sequence contract) about any other process?

Or must you know just a little about that process, and that process? But not aware of the costs? That life is at stake?

Note

I have written something about this in a paper, where I tried to define a new term. I called it the "Office Mapping Factor" [1]. A friend computer scientist said it was the "coolest" thing I had done. (He might not be aware of this or this.) Here is the abstract of [1]:

This case observation describes how an embedded industrial software architecture was “mapped” onto an office layout. It describes a particular type of program architecture that does this mapping rather well. The more a programmer knows what to do, and so may withdraw to his office and do it, the higher the cohesion or completeness. The less s/he has to know about what is going on in other offices, the lower the coupling or disturbance. The project, which made us aware of this, was an embedded system built on the well-known process data-flow architecture. All interprocess communication that carried data was on synchronous, blocking channels. In this programming paradigm, it is possible for a process to refuse to “listen” on a channel while it is busy doing other things. We think that this in a way corresponds to closing the door to an office. When another process needs to communicate with such a process, it will simply be blocked (and descheduled). No queuing is done. The process, or the programmer, need not worry about holding up others. The net result seems to be good isolation of work and easier implementation. The isolation also enables faster pinpointing of where an error may be and, hence, in fixing the error in one place only. Even before the product was shipped, it was possible to keep the system with close to zero known errors. The paradigm described here has become a valuable tool in our toolbox. However, when this paradigm is used, one must also pay attention should complexity start to grow beyond expectations, as it may be a sign of too high cohesion or too little coupling.

I will not repeat [1] here, but I see no reason to rephrase this, from chapter 3 point 1:

For a process, not being able to control when it is going to be used by the other processes. Serving “in between” or putting calls in a local queue makes it much more complicated to have “quality cohesion”. [Java classes, for example, cannot prevent their methods being called (except, through ‘synchronized’, where other synchronized methods can't run if one is already running). But that does not prevent the method from happening. It just delays it. It cannot be stopped, even if the object is not in a position to service it (like a "get" on an empty buffer). Not being able to control when it may be used by other processes means that things of importance to a process may change without it being aware of it. This trait is further discussed here, since the other points (below) only to a small degree are valid in our project.]

I also have written about WYSIWYG semantics at my note 007 - Synchronous and asynchronous.

Origins

The term WYSIWYG semantics seems to stem from [2]:

One crucial benefit of CSP is that its thread semantics are compositional (i.e. WYSIWYG), whereas monitor thread semantics are context-sensitive (i.e. non-WYSIWYG and that's why they hurt!). Example: to write and understand one synchronized method in a (Java) class, we need to write and understand all the synchronized methods in that class at the same time -- we can't knock them off one-by-one! This does not scale!! We have a combinatorial explosion of complexity!!!

With CSP, threads can refuse individual events if they are not in a state to accept them. They do not wait on shared condition variables and do not rely on other methods to fix things up for them. Each method has its own contract and looks after itself. This type of logic does scale.

Practical example 1: A server having a session with one of several clients

There are five clients, named A-E and one server S. Helper process for S is called H

Server process S (easy case)

Client A send a message service request to S
S needs services from H, so this will take a while
S waits for response from H in a select
S receives response from H (from nowhere else)
S sends reply back to client A
S listens in a fair way on client channels (pipes etc.), and receives a request from client B

This is WYSIWYG semantics.

A possible architecture could be one point-to-point channel (or p-p pipes) between each client and the server, and two p-p channels between S and H (request and response).

Server A could have done something else in the meantime, it didn't have to just wait for reply from S. This could have been more or less easy, see later.

Server process S (difficult case)

Client A send a message service request to S

S needs services from H, so this will take a while
S waits for response from H in a select
S receives instead of response from H, another service request from client B
S pushes this request away (luckily this was the only request)
S finally receives response from H
S sends reply back to client A
S pops request from client B and processes it

In this case, server S needs to be its own "scheduler", by pushing and popping requests. Lucky that it was in a state where this was ok. It didn't need to. This architecture is difficult to program. Maybe the programmer "knew" that the other clients would not be active, so it needed only push one request. When a new programmer sees this standard client / server model, it does not "know" that the client only has a one position stack. This is discovered by inspecting a log after having installed the system at 1000 sites. The server had crashed, and a service man had managed to get a log.

This is not WYSIWYG semantics.

An architecture that could cause this to happen is a common message queue that the run-time system uses to pick new messages from, to the server. At any time, any client could send requests into this common queue.

Client processes

So far we have not said anything about the client processes. Does it make any difference if they block or send and forget the request into (an infinite) queue?

A WYSIWYG scenario: If the first request blocks immediately (zero buffered channel), and the server not at this channel yet or in fact is at this channel, then the request - reply sequence will be "atomic" as seen from the outside. If several clients all try to send a request at the same time: still as "atomic". We can knock them off one-by-one! This scales!

If the client sends into a buffered channel (see note 034), it's still WYSIWYG, since the channel pipe disregarding size is process to process. Not shared. Even the Any2OneChannel of JCSP [3] will be fine, since:

Any2OneChannel is an interface for a channel which is safe for use by many writing processes but only one reader. Writing processes compete with each other to use the channel. Only the reader and one writer will actually be using the channel at any one time. This is managed by the channel – user processes just read from or write to it.

Please note that this is a safely shared channel and not a message gatherer. Currently, gathering has to be managed by writing an active process.

What WYSIWYG does not cover

I don't think that this term would cover problems like race conditions, hazards, deadlock, livelock, starvation, pointer errors or anything causing "segment fault". Even if one do get something else than you see, to me the term What You See Is What You Get semantics would not cover these matters.

They belong to different "error spaces".

References

[1] - "High Cohesion and Low Coupling: the Office Mapping Factor". Read at http://www.teigfam.net/oyvind/pub/pub_details.html#TheOfficeMappingFactor

[2] - Letter to Edward A. Parrish, The Editor, IEEE Computer. Peter Welch (University of Kent, UK) et al. (Dead url: http://www.cs.bris.ac.uk/~alan/Java/ieeelet.html (1997)). Internet archive at http://web.archive.org/web/19991013044050/http://www.cs.bris.ac.uk/~alan/Java/ieeelet.html (1999)

[3] - org.jcsp.lang Interface Any2OneChannel, see http://www.cs.kent.ac.uk/projects/ofa/jcsp/jcsp-1.1-rc4/jcsp-doc/org/jcsp/lang/Any2OneChannel.html (part of Communicating Sequential Processes for Java^TM (JCSP) at http://www.cs.kent.ac.uk/projects/ofa/jcsp/)

.
.

035 - Channels and rendezvous vs. safety-critical systems

2011-12-19T21:07:00.014+01:00

In this note I will try to figure out why the Ada programming language (in the Ada Ravenscar Profile [1]) had to make a subset for safety-critical systems that explicitly prohibited the beautiful select. And - would occam (or perhaps CSP - but this has no meaning, since CSP is an algebra) - have had to do the same, i.e. prohibit ALTs (and even CHANnels?).

To it me it seems like the Ada language has an inherent problem: the select mechanism and the rendezvous are based on queuing. I was surprised to read about this, as I have learnt that the Ada concurrency features are based on CSP. I will try to explain how this understanding could be plausible.

I have done some CSP channel scheduler coding, so my fingers are dirty [3]. There is no queue in what we did. There was no queue in the transputer which ran occam natively. There was no queue in SPoC (I have written some about the Southampton Portable occam Compiler here).

When a receiver sets up an input ALT, all the components' channels may be blocked upon by several senders. They have all flagged their sender channel as "first". There is no queue of these senders. If more that one is ready when the input ALT is started, either the first seen (PRI ALT) or a non-deterministic choice should be made. (Implementation may be lacking this feature, but verifiers like Promela or CSP will explore all possible choices when they hunt for liveness properties.) Also, none of the senders may be ready yet.

Instead of queuing the senders, the senders' channels are tagged as "first" on channel, and the sender is left (blocked or descheduled). It will not see light again before a receiver (in an ALT or just an input) also appears on the channel, as "second". So, there is no need to queue anything. But, is the "implicit queue" a proper queue, so that there is no difference?

I am not sure why Ada does queuing on the select, accept and entry mechanism. Each entry has a queue of processes, fifo by time sequence. Is it because it also supports protected objects (classical monitor with guards instead of condition variables)? Is it because it's bidirectional?

The Ravenscar profile prohibits select, demands protected objects, but queue size is 1, which to me seems to collapse the queue to no queue. From [4] one of the Ravenscar features is described as "Protected types and objects, defined at the library level, with a maximum of one entry per object and with a maximum of one task queued at any time on that entry. The entry barrier must be a single Boolean variable (or a Boolean literal)."

So, oversimplified (or wrong?), to make Ada usable for safety critical systems, it seems to have collapsed to occam?

Of course, high integrity Ada as defined by the Ravenscar profile, is concerned with schedulability analysis, static analysis, control flow, data flow, information flow, symbolic execution and formal code verification.

And at the bottom line, "the definition of Ravenscar requires preemptive scheduling of tasks." But seen from an occam perspective, the subsequent sentence is logical: "However a similar profile could be defined that specified non-preemptive execution" - but from the Ravenscar profile, I don't understand that differentiation. I would understand the latter better than the first, from schedulability perspective. Of course, in a cooperative scheduling perspective, a single process must run to next scheduling point before another process is allowed to (or ru..n with any number of dots, up to indefinite), whereas with preemptive, the next process could be there in a snap. But that's most interesting if hardware driven software interrupts are not considered. Occam didn't have interrupts, they were proper processes with a channel placed at an event pin.

The ALT or select mechanism is the non-deterministic feature of these languages. I found a paper for a conference, where they said that unwanted non-determinism is not acceptable (causing races etc.), but controlled non-determinism is something to allow, or even strive for [5].

So, the Ada Ravenscar profile, also makes controlled non-determinism unwanted?. Did they really have to? Promela expands all the executable paths of its case structure, what's wrong with Ada not to allow it?

Should you stumble upon this note and have some comments, please comment or send me a mail!

A comment

I have received this comment in an email from a computer scientist (in reply to my direct mail query):

"Ada tasks can communicate in two ways - via message-passing (rendezvous) or via protected objects. The latter decouples the tasks more and allows more accurate timing (scheduling) analysis to be carried put. The Ravenscar profile is for hard real-time systems where schedulability is crucial. So the profile, in effect, throws away the rendezvous and only allows communication and synchronisation via protected obejcts.

Now for occam there is no alternative to message passing and the rendezvous - so it is not easy to see how an occam-ravenscar profile could be defined."

Later on (my italics):

"I had a quick read of your article, I still feel you are not being fair to Ada - it supports, as I said, two quite different communication methods - message passing and monitor.

For message passing it uses a many-to-one model - occam uses one-to-one, so queuing is more an issue in the Ada model (though queuing on ALTs takes the place of queuing on entries - Ada programs need less ALTs/selects)

Now Ravenscar has removed 'select' as it has removed message-based communication, but the protected object is still a source of (good) non-determinacy. Many tasks calling the same PO will access in an order not predetermined."

Thank you for these comments, and for your time!

References

[1] - The Ada Ravenscar Profile: http://en.wikipedia.org/wiki/Ravenscar_profile

[2] - Occam programming language: http://en.wikipedia.org/wiki/Occam_(programming_language)

[3] - "New ALT for Application Timers and Synchronisation Point Scheduling", (Øyvind Teig, Per Johan vannebo), at http://www.teigfam.net/oyvind/pub/pub_details.html#NewALT

[4] -"Guide for the use of the Ada Ravenscar Profile in high integrity systems", Alan Burns, Brian Dobbing and Tullio Vardanega, at http://www.sigada.org/ada_letters/jun2004/ravenscar_article.pdf

[5] - Will find that reference one day. I may have it at work.
.

034 - Output guard vs. "channel ready" channel

2011-12-03T11:41:00.039+01:00

Last edit: 3.Jan.2012

On the airplane home, the other day, I came upon an idea that I thought would make life easier. When I told a guy at work, he said, "oh, it's already in Linux".

I assume that you're somewhat familiar with "channels". In this respect I was raised with the occam language. (You would find Wikipedia-articles for most of the names I'll throw here. So I won't get too deep into some details.)

Here, a channel is based on CSP (later), and it is one-directional, non-buffered and blocking. It's even one-to-one. It connects tight processes to each other. It also allows threads to be equally tight. So, the first process that comes to a channel (be it sender or receiver), must be descheduled and not ever run again before the second process arrives on the channel. At that time there's a memcpy of the correct data from the sender's private context, to the receiver's matching field. It is possible to make it type safe. The message is passed across, while both sender and receiver are still. In Ada it's called a rendezvous. In Go it has no name. When the memcpy is done, usually the second on the channel just runs on, while the first is put in the ready-to-be-scheduled queue.

There is a mechanism called on ALT or a "select", where a process may wait for a number of (input) channels. A channel may be with data, without data, or timeouts. In my world (opposite to Linux select), there is exactly one cause that may trigger the SELECT. I will use capitalized "SELECT" here to cover any such idiom, to differentiate from the Linux "select". The first cause that triggers the SELECT will be stored and cause a later rescheduling, but any potential causes that "lost", will not be allowed to cause an additional triggering. Only one cause per scheduling of a process. Very different from Linux select.

Said differently: the SELECT will put all the channels in the set to "first was here". When the first "second" contender on a channel in the set arrives, all the "first" attributes are removed from the rest of the set. This is a beautiful algorithm, that I think was first implemented in microcode on the first transputer processer in 1984. Inmos in Bristol, UK, made it, and David May and Tony Hoare (the father of CSP, "Communicating Sequential Processes" process algebra) were the main architects. The transputer basically was a machine designed to run the subset of CSP that was called occam. It's 28 years ago. Ada based its concurrency model on CSP. And 26 years after occam, Google designed the Go programming language, which also has based its concurrency model on CSP, just like the same guys had done at Lucent some 10 years before with their Limbo language. Except, at that time they didn't state the roots of their "chan of protocol". And transputer designer David May now is active with a company called XMOS, and a language called XC, based on the same ideas. So, CSP-based concurrency seems to be increasingly coming.

With this scheme communication equals synchronization.

In CSP they talk about "choice". It may be deterministic or "external" or nondeterministic or "internal" (often written as [] and |~|).

The SELECT is for the teacher who assigns work to his students. He then falls to sleep (or does other things) and is awaked when the first student is finished. The student is then served, and the SELECT is entered again. The new set may include the first student, when some fair scheme is used to avoid him spam the teacher. Or the SELECT may be non-deterministic. Read in the literature, f.ex. the Promela Wiki-article (more below).

With this scheme, the teacher may say that one of the students is not allowed to deliver his result before after two hours. So, there may be timers in the SELECT set. This would also make busy polling possible, should this be needed at the application level. But the run-time scheduler NEVER needs busy-polling. Another way to say this is that a process always has a cause when it's scheduled. Just think about Java's notify and notifyall, which may schedule a process for no reason!

Also, there may be an INPUT GUARD for each channel (applied to our special student for two hours). This is usually implemented as a conditional to each member of the SELECT set. Think of the SELECT GUARD set as a bit-mask, one bit per element in the set, just like in Linux select.

Now I'm getting closer to the theme of this note: Output guard vs. "channel ready" channel.

Some times it's required to buffer a message on a channel. In occam we used to do this by inserting a single buffer process, or a couple of processes to make one composite process called an "overflow buffer", which made it possible to detect a too eager producer and associated too slow consumer - at this application level. One time I needed a buffer of a hundred buffer positions, and I just started 100 buffers. Easy! The downside was that there would be 101 memory copies, so most often a ring buffer was administered inside the first of the two processes in the composite overflow buffer process.

The alternative could be for the eager producer (it has no idea itself how eager it is) to have something called an OUTPUT GUARD available. If you want to see it, have a look at the Promela process meta language (Wikipedia). This way, both inputs and outputs may be collected in the SELECT. So, having just one channel, plus an "else", then the potential receiver, if it is able to receive, will inform through the output guard if it's ok to send. If not, the else will be taken. The SELECT with both input and output looks like the Linux select, but that one doesn't have the synchronized mechanism in the bottom.

It's a nice mechanism. I have never had it in my world. It was not implemented on the transputer, since it was a multi-core programming language. And implementing it across a network (links) presumably was not worth it. There was another mechanism to get the same functionality. If you know what mechanism, please comment.

An output guard mechanism is good for detecting a broken line. To detected a connected line, there would have to be polling. Busy polling is most often necessary for this (assuming no hw edge interrupt). But the output guard seems not so relevant to handle the fast producer, slow consumer scenario - between internal processes. The same broken / connected scenario also would go for detecting that the other process has stopped, for some reason. Perhaps enough for a new blog note here.

So, here's the solution, to avoid the output guard, to avoid busy polling. The solution that also almost exists in Linux.

A standard channel has zero buffers. Have a look at Promela and Go. They both support buffered channels. So, why not add them to [1]? What should happen when such a channel is full? The channel send just returns a flag saying that it’s full, and the channel would not block in that situation. Observe that blocking is perfectly ok and does not impact what a system is able to do. With enough "parallel slackness" to get all the I/O going at required speed, having a process block (waiting in a non-busy way, for the second to arrive at the channel). But for a driver, close to the "asynchronous world" it's nice not to block to get rid of incoming messages. And it's nice not to send it into an asynchronous message system that may plainly overflow, crash and restart. We need a better system. We need the driver process to not block in that situation and itself handle overflows.

So, in the suggestion, after the return of the buffer full flag, the process knows this and if any incoming message arrives from the external asynchronous world, it may set a hold line, send a hold message or throw away the message - or whatever it decides to. It's up to the process.

In this state, the process will SELECT around a timeout channel and this channel that's triggered by the run-time system, that arrives when there is room in the channel. The receiver has picked out one message or all, that semantics is not so important here.

When this "channel ready" channel fires, the process can send off the last message, which may also be tagged with an overflow bit, on the channel. And it's guaranteed to succeed (unlike in Linux).

I was rather satisfied by inventing this on the plane home. I didn't like that it was already in Linux write and select. I have tried to describe some of the dissimilarities. See [2] for write, EAGAIN or EWOULDBLOCK and [3] for select (or pselect) and fd_set *writefds. It's in the context of writing to a Linux pipe.

But please, could anyone help me with describing how my solution is much "thread safer" than Linux? I need more than this:

I have more of a language, since I would do as described and reasoned above. Linux calls and Linux processes and threads are not understood by C or C++. Up to C++11 at least. And only so much understood by Linux. A C++ library for Linux needed several mutexes and semaphors to implement an ALT/SELECT in the CSP context [4]. So, are write and select usages in Linux really thread safe? I have a scheduler that is driven by the channels, and it's not preemptive. And why do Linux programmers brag that they "program single-threaded"? Is it an indication that the Linux process model is too coarse or difficult or expensive? So that they haven't learned to decompose into processes, when there is state or roles to hide away into a separate process? After object-orientation or OO, it's time for process-orientation as an additional idiom.

Should have been fun to set up a comparison table of the CSP type SELECT with channel ready channel and the similar Linux solution. Even if they are both fruits, I fear that one is apple and the other bananas?

A Linux pipe vs. channel trait

Observe that a Linux pipe is byte-wise. You send so many bytes and then so many bytes, and they are concatenated in the pipe. There has to be a common view of how to pick out each message or token.

This implies that you could read past a token, since a sender may have to break up a message into several chunks. There is no foldback, you cannot push anything back again. So, the receiver has to keep track of these things.

A channel is message based. A full message per chunk. Finito.

Common usage of Linux select

A collegue told me that the common usage of a Linux select is to remove active polling for events from remote machines from the application level, in say a http server (router). He said that he always used select to wait for events on already open sockets. One socket per bit in the select bit map.

Internally between threads he never had used select.

And he rarely used ip addresses internally between processes, so no select there either.

"My idea" - again

I found an interesting sentence in [5]. Here is its process-oriented specification of a buffer: If a buffer is empty it must be ready for input. If it contains some messages, then the buffer must be ready to output the next message required. It is also possible that it will accept further input, but it does not have to. (my comment: is this buffer full?) But here is the interesting sentence: "If further events are to be possible (such as a channel which can report on whether or not the channel is empty), then...". This is "my" channel! I am even more convinced now, that "my" idea was quite good!

---

[1] - http://www.teigfam.net/oyvind/pub/pub_details.html#NewALT
"New ALT for Application Timers and Synchronisation Point Scheduling"
Øyvind Teig and Per Johan Vannebo
Communicating Process Architectures 2009 (CPA-2009)
Peter H. Welch et. al. (Eds.)
IOS Press, 2009, ISBN 978-1-60750-065-0

[2] - http://linux.die.net/man/2/write

[3] - http://linux.die.net/man/2/select

[4] - Search for "C++CSP2"

[5] - Concurrent and real-time systems. The CSP approach. Steve Schneider. Wiley, 2000. Page 210, example about buffers (7.4.1).
.
.

030 - Burning an mp3 music CD from iTunes

2011-09-26T21:21:00.002+02:00

First written 22.Aug.2011, updated 24.Aug.11. iTunes 10.4, Mac OS Snow Leopard 10.6.8.

Two entries for some songs, and a repair action

First: The normal no-fuzz procedure is described at http://www.wikihow.com/Burn-iTunes-Music-to-MP3-CD-for-Stereos-and-CD-Players

I burnt an mp3 music CD the other day. Since some of the songs were not in mp3 format, I had to manually convert them to mp3 before I burnt the playlist. And I had to export the playlist to a file and then edit it in a text editor to replace all occurances of .m4a with .mp3. Then I had to import the playlist-file again. Only then did iTunes obey my wish to burn only .mp3 files, and not complain that "81 songs", sorry, could not be burnt.

When I afterwards looked at one of the original CDs in the Music folder, it had two entries for each song: one m4a and one mp3, or one mp3 and another list entry also as mp3! And playing that album, every song was played twice.

I then used Finder to track the music folder and move the mp3 files, that really should not be there, after the mp3 CD had been burnt, into another directory outside iTunes. Then I in iTunes scrolled through the list in the album, and it showed an exclamation mark for every song that was gone. Then I in iTunes picked them all out and moved them to the bin. The figure above shows the an album in the Music folder (left) and the almost same album as seen in the playlist.

mp3 CD in 2011?

Everybody else made mp3 CDs ten years ago. But it's only now that we in fact have two units that play them! The car stereo and the Denon CEOL. The second doesn't need mp3 CDs since it streams. And the car radio also has the aux input that takes music from iPad or iPhone's headphone outputs (see blog 028). But a few mp3 CDs in the car doesn't hurt, and makes it possible to see the song names in the car's display, and control music playing by steering wheel buttons. And since there's no USB memory stick input that would have obsoleted the mp3 CD, we will have to use standard CD, mp3 CD or aux input.

I wanted to extract the mp3 CD from our iTunes music collection.

Some more points

Some mp3 songs also all of a sudden appeared twice in the iTunes list, pointing to the same mp3 song. One lits entry has to go, so delete it, but do answer "yes" to the question whether you want to keep the file. I didn't realize and did kill a song or two. Some day I'll learn when I try to play it. Then it's going to be good with the synched backup I have!

If an album comes up with exclamation marks just by itself, it's only the list entry that's been added. Delete the exclamation mark lines only, since the mp3 file in itself is not repeated. Two files can't have the same name in the same directory.

Don't they want me to make mp3 CDs?

Probably not! Am I not supposed to make mp3 of purchased music? I think it's only ripped CDs that may be stored as mp3 out of the box, provided I had imported them to mp3.

I still haven't finished the clean-up! Some 40 albums take time! I really feel stupid.

Advice

Don't convert to mp3 if you want to burn an mp3 CD. Take the onces it has for you. Be sad and tell Apple.

But is there another way?

If there is something really wrong with what I did wrong, please comment! Like, if it's so wrong and it would have been easy to get it right, had I hit the right action trail immediately. Right?

Reported to http://www.apple.com/feedback/itunesapp.html.
.

029 - From ceramic vase to cutlery drainer

2011-09-26T21:17:00.002+02:00

Moved to http://oyvteig-2.blogspot.com/2011/09/029-from-ceramic-vase-to-cutlery.html
.

028 - DIY prototype of an iPad 2 holder for car glove compartment

2011-08-10T23:02:00.013+02:00

Moved to http://oyvteig-2.blogspot.com/2011/09/028-diy-prototype-of-ipad-2-holder-for.html
.

027 - Experiencing Apple AirPlay

2011-07-19T21:40:00.005+02:00

Referring to blog 019 - we ended up with a Denon RCD N7 Network CD Receiver (CEOL). I gave up on Sonos. I got real AirPlay instead. Now we can listen to music at places and by combinations I hadn't read myself to know. Let me go on from blog 019:

Denon RCD N7 (CEOL)

This was bought at Hi-Fi klubben here in Trondheim in July 2011. Bringing it home, the first non-power cable I connected was the home network's ethernet cable. Joy! The only cables I plugged out from the 20 years old Denon Precision Audio Component / AM-FM Stereo Receiver DRA-335R were analogue. It has been tired some years now, together with the Denon PCM Audio Technology / Compact Disc Player DCD-680. Both were also bought at HiFiKlubben back then, and will now be stacked on the loft. Hard to scrap a friend.

The Ceol took charge and asked me if I wanted to update its firmware. Yes, I did - and it did the job in some 30 minutes.

I had bought the thing for its AirPlay capability (there are plenty other nice boxes without), and went to http://www.denon.eu/airplay/en/. First I had to pick out our unit's serial number, MAC address and upgrade ID. Paying with PayPal has always been a pleasure, so I expected the Ceol to be my everything's remote speaker any minute. For 48€ I thought it should. It didn't. Even if I got a receipt that all was fine - and in the background: Apple had got their licence fee. Since I get a salary every month I don't protest if others want some, even if they put it in the bank.

The Ceol told me I could upgrade with AirPort, ok. I pushed the right menu entries, and it said Authenticating for a minute, an hour, a night. All it accepts then is a power off, so I retried. Nope. A letter to HiFiKlubben and a reply from support adviced me to restart to factory settings. The updated firmware and the 48€ investment should be fine. In the manual it's called "Resetting the microprocessor: unplug the power cord, then hold volume up and down simultaneously and plug in the cable, then release the buttons". However, I used the slightly different procedure in the support email: "Total reset (to factory settings out of the box / virgin mode) 1. Switch Off the unit and unplug mains plug from wall outlet. 2. Press and hold both Volume up and Cursor down buttons on the front panel and plug the mains plug into the wall outlet. 3. Release the two buttons."

This worked perfectly (as probably would have the manual's recipe?): AirPlay soon was in the box. And everywhere! Running Apple's Remote app on the iPad now, the iTunes library comes up as usual, but both the Ceol and the computer come up! I get "surround sound" that I did't know. The tv speakers with nice bass sound (from JackOSX router on the Mac Mini, see blog 019) also appeared. And they are in phase (thanks to some clock synchronization messages in the AirPlay protocol I believe).

Now Gary Moore's Ballads & Blues 1982-1994 plays like music has never been heard in this house. At the moment: bliss.

Experience

After a month's use I'd say I'm pretty satisfied with AirPlay. But there are some repeating weak spots:

Sound level at connection.
I still haven't figured out why it comes on with an enormous volume some times. So, if the amp is more powerful than the speakers, the AirPlay start-up volume may destroy your speakers! For me, it's ok, though.
Sound level up when down or down when up
The app volume control and the real volume may be out of synch before I have touch the app's volume control. So, when I try to push the volume down (from a shown high volume), it increases (from the unit's low volume) - so that the overall effect is increased volume! Aften that up is up and down is down.
Broken connection.
Some times one of the AirPlay "speakers" fall out, and I'd have to manually reconnect.
Speakers out of phase.
One time, while iTunes (version 10.4) converted a bunch of songs to mp3 (and was very busy in one of its "threads"), and it was playing a song simultaneously - the speakers went out of synch. I know this, since it was easy to hear. I would guess 50 ms since it was not that much.
"Connecting to AirPlay unit" when envisaged as already connected.
Every time I find a new melody it seems to disconnect and then needs some extra time to reconnect. I experience this as not necessary.
Outdated display.
I have seen Denon not display the present melody, when I played iTunes through the Remote app. Didn't iTunes send it, or didn't Denon update? It stayed with the other day's melody through a whole playing session. At the moment I dont't know what updates it.

Reported to http://www.apple.com/feedback/itunesapp.html. I have found no similar feedback link to Denon.
.

026 - iOS picture sorting and file system do not tango

2011-07-02T10:37:00.020+02:00

(2 July 2011, updated 24 October 2011)

This note discusses synching pictures on iOS units (iDevices) by letting iTunes use a disk directory for picture synching. It does not initially deal with having iTunes synch using iPhoto, but I plan to also handle that. I use Mac OS X Snow Leopard. I will update this note as time goes. If it's not been updated, it's probably because I haven't seen anything new.

iOS sorting by modification date only

Update 24.Oct.2011 with iOS 5.0: Apple seem to have gone back to alphabetical sorting again! However, in view of what I have written below, maybe they now have combined sorting methods, as I have suggested?

Apple has decided that the iPad, iPhone or iPod (all running iOS) pictures and movies shall be sorted according to last modification date and time. It didn't start that way, but has been like that since iOS 4.

I would perhaps want iOS to let me select sorting method. Letting me choose alphabetical sorting could have been great, provided it worked. Since the alphabet I use is sortable (!?), it's a nice feature that our culture has become accustomed to. However, since camera picture names contain a number, the series would sooner or later wrap around: DSC_9999 would become DSC_0000. So any number would repeat, making alphabetical sorting difficult. And no matter how much time increases, it only seems to wrap at each big bang(?). (Being an embedded programmer I know that having a finite word length, counting the number of system clock tics does indeed overflow to zero. But that's why we're there, to know about these things.)

Maybe combining some date/time and alphabetical sorting would have been smart? The engineers at Apple of course know that being smart some times is not always that smart. The framework in which to be smart may change, and in the new context it wasn't smart any more.

So, sorting photos by date/time has come to stay. I'll have to take it in.

Vacation pictures oops.. changed!

If I change a file, cropping it, resizing it, change the light or anything, then it would now cause that picture to get today's modification date/time. Synching it with iTunes to an iOS unit would place it down at the bottom in the sorted list.

So I need to overwrite modification date by creation date

I found an AppleScript that does exactly this on the net [1]. I'll call it "TheScript" from now on. Perfect, now the photos are in lane again after the next iTunes synch also. But hey, finding it, downloading it and start to use it is not simple in any context, Apple?

Say the disk directory where iTunes finds the pictures is on a local disk, and say that TimeMachine is used for backup, then all is fine.

However, if the picture directory is on a network disk, or if as above without TimeMachine as backup, or if not only backup but also synchronization is used, then I may have a problem.

My synch program may push my pictures out of lane again!

Synching disks and synching iOS units with iTunes are not the same thing. I'll try to be precise below. Later in 2011 Apple iCloud synching will happen without iTunes.

Skip this chapter if you always plan to run TheScript before you synchronize the disk with another disk, after having modified a file.

To get disk synchronization to work, modifying a set of files should be "atomic" with respect to disk synching. So, make yourself non-disturbable in that phase. A modified file should not be synched before TheScript!

I'll try to explain the situation that might arise if you modified a file, got disturbed, and then synched the disks "out of synch".

I use an old friend called ChronoSync. I have used it for years to synch several machines, using a network disk as the base. Whenever I change a photo, I do it at some local machine and synch the disks and then let iTunes use the synched base at the network disk for its photo synching.

However, since I have modified a photo and thus made it newer, ChronoSync will (if I run it in this phase) let the newest version survive. Later iTunes iOS synching will place the picture at the bottom.

I have found no way for the present ChronoSync to work this out. A change could be something like this, to be used after file change and we don't have TheScript:

Resolve conflict if two files have equal creation date to use the newest modified file but then change modification dates of both to become creation date

Or, how about a special synch session called iOS picture/movie synching?

I have informed Econ Technologies about this 2July11 (http://www.econtechnologies.com//pages/support/support_comment.php). (And they have replied, stay tuned)

Back to iTunes: re-date alphanumerically sorted files?

I have no way to solve this! The name of all my files have started with som number and then a sentence, like "314 pythagoras.jpg" and "271 e.jpg". The operating system understands that the characters '0'-'9' are treated as numbers and sorted accordingly. I have called this alphabetical sorting, even if it does understand fields. So the 271 file is sorted as the first.

Some times I could learn AppleScript and write a script that solves my problem: keep dates as much as possible, understand modification date and spread the set out over the dates wanted. Sorting by pushing around pictures by hand is perhaps best.. But this is difficult!

Inserting a picture or a movie into the iOS sorting sequence

Some times you need to add a picture into a sequence. Maybe you want a poster in front of each section, or the next country's flag. This is almost easy: 1.) Make a copy of the neighboring file. This has the same creation date. Experiment with the file before or after, it's not obvious how equal date/times are sorted. 2.) Open the new file in an editor. I use Graphic Convertor. 3.) Paste the image you have made prior to this so that it's inserted into the just opened picture, so that all is overwritten. You may have to delete all contents first. 4.) Save the file with the new contents. 5.) Run TheScript to make modification date from creation date.

This works for picture files.

Movies like .mov I have not been able to fix yet. Stay tuned.

Refs

[1] - MacFileDatesChanger by Daniel A. Shockley, http://www.danshockley.com. Includes some code from Paul Berkowitz and Nigel Garvey on applescript-users mailing list.

.

025 - iPad2 on vacation, ups and downs

2011-06-13T12:32:00.043+02:00

This blog lists some random experience with using an iPad2 in Italy in the summer of 2011. The iOS version was 4.3.3. Some if these experiences may be of interest to others..

1 - Painting from a picture with a grid ::

I did try paining water colour (aquarelle) again on this visit. Sitting at the spot to draw and paint feels most artistic. However, as the light and shadows, and indeed colours change, it's easy to shoot a picture and use it to paint off-spot.

I took the picture and imported it to the iPad. More about this import scheme later; it does have some peculiar functionality.

It's so much easier to draw the picture to paper if we have a grid. A long arm and the thumb on a pencil is difficult with and iPad. With the iPad and its 4/3 ratio screen, a 4x3 grid is enough. I did not succeed making this in the iPad while the paint was wet, so I taped dental floss on the iPad! It worked, but was not very practical when touch was again needed.

When the paint had dried and I had better time, I found a way to overlay a grid onto a picture, on the iPad - while on vacation:

1. In Apple Pages, make a 4x3 table, with size 16 x 12 cm or as close as you can get. Save it. Turn the iPad to landscape and make the table as big as possible.

2. Take s screen shot (Home and On/Off buttons simultaneously). It is saved in Camera Roll.

3. Open ArtStudio. Load this screen cut from Camera Roll. Crop it. Rescale it to 2048x1536 since this is the internal format that ArtStudio "likes" best. Don't keep aspect ratio, as the table you made in Pages may not be accurately 4/3. Now smart-select (use magic wand) one square (512x512) and clear the contents. This makes it transparent. Do this 12 times. Now you have a 4x3 transparent grid. Save it to the Camera Roll.

4. In ArtStudio, load the picture you want to paint. Make a new layer, and insert the just saved picture in that layer. Save it. Congratulations, you don't need the dental floss any more for this stuff!

I have informed Lucky Clan (http://www.iphoneclan.com/artstudio/) about this - as having a built-in function would have been much nicer. I have also informed Apple (http://www.apple.com/feedback/iphone.html), as a magical "display picture with grid" would have been great! Or maybe there already is an app for this? I have not found any way for Pages to put pictures as table background.

By the way, the picture is from San Gimignano. Also, I did not want to transform the picture inside ArtStudio to make a computer water colour. Would have been nice, but not this time.

2 - Deleting imported pictures from iPad but wanting to keep them in the camera ::

This problem is as simple to describe as the iOS solution is too simple.

The first night we imported the first day's new pictures from the camera. Some were good, some were deleted. We had created one first day photo event. ("hendelse" in Norwegian.) Good!
The next day we imported the second day's pictures from the camera. We were asked if we wanted to import the duplicates. Good. We answered no. We had one second day event. Good!
But the pictures we deleted day one were put into a second first day event. This is not what we wanted!
The third day I understood the problem: it's not possible to delete on the iPad alone! If a picture is deleted from the iPad, it must also be deleted from the camera. However, since that picture had indeed been taken, and today's cameras have enough memory to keep everything for a week or four, having a backup of iPad's deleted pictures in the camera would have been nice.

There could be several ways to implement this. I have some ideas, but I trust Apple to solve this.

Only observe, there could be (like) two cameras to import from. An additional problem is then if there should be(like) two first day events or one joined?

Reported to http://www.apple.com/feedback/ipad.html.

3 - Using the iPad Navigon app for navigation in the car ::

I have informed a Navigon support mail address about this blog.

Roundabout

We were using Navigon 1.8.2.

This picture shows a typical problem we discovered. We're not certain who's wrong here, but we got it wrong. Here's the route:

Via Pisana, 5**** Poggibonsi → Via San Gimignano, 53036 Poggibonsi
navigonITA://route/?target=address//ITA/5****/POGGIBONSI/VIA%20PISANA//11.13664/43.47654&
target=address//ITA/53036/POGGIBONSI/VIA%20SAN%20GIMIGNANO//11.12998/43.47651

I have isolated the problem to around this roundabout. The routing is fine, but starting from the green flag the voice tells us to:

First take the second road in the roundabout
Then take the third road in the roundabout

We did 1. as told and went up the yellow road. It discovered we were wrong and correctly took us back to the roundabout coming in on Via Galileo Galilei, the white road. It then told us to take the second road, which was correct. But it still asked us to take the third in the final roundabout.

If Navigon is right, then it did not count the white road coming into the first roundabout - but it did count the north white road in the second.

Disclaimer: I don't really know if the map corresponds 100% with the roads, but I think so.

Only going to first entry of route?

I think a Navigon "route" is a list of targets that we can only jump into at one place: the first entry. So, if you have made San Gimignano, no special street, the first entry in the route nad want to go from "here" there is a danger that you'll use 15 minutes trying to reach the first entry from here, like telling Navigon to to go from home to some neighbour. It's made for this kind of stuff, but the Navigon people could probably detect this, and ask me if I wanted to go to the first or second entry in the route. And then next time, I could tell it to go from here to the third entry. And then I would not have to cut off the next goal once I've got there. Maybe this is not what Navigator software is about, maybe others have this, and maybe even Navigon has it? Bare with me: I am new to this!

Map errors

There were obvious map errors, mosty associated with temporary changes. NAVTEQ certainly has some work to do there.

Help

Navigon contains no help whatsoever(?). Touch is not logical, it has to be learnt. Swiping! One, two, three or even more fingers. There is some common reaction to them among apps, but I'd certainly like to know from day one, or minute one, how they are used by Navigon. Tip for Navigon: download ArtStudio. They have a single help screen called "Quick tutorial" and courses called "Lessons".

Stability

Some times Navigon crashed and we had to restart, like once every two hours. It came back to where we were, but I certainly hope that Navigon has asked iOS to ask me to send crash logs back to them. Once simulation hung and Navigon did not respond. What I then did was terrible, next chapter.

Post script: another experience with Navigon

Have a look at this note: 028 - DIY prototype of an iPad 2 holder for car glove compartment. The bottom chapter there also comments on Navigon and the car's built-in navgiation.

4 - Too easy to fatally remove an iOS app, when only stopping it was the intention? ::

I deleted Navigon, the navigation software that we relied on, with my finger one late night. When I switched in the brain, it was too late. Norwegian iPad screens, but you should reckognize them ok:

Then I decided that this was partly Apple's fault, even if I hadn't done my homework (stay tuned).

I was tired and pressed buttons in a frenzy when my fingers took the wrong path between wobbling icon with a red minus (top in figure, to kill or stop a program) and a wobbling icon black cross (bottom in figure, to remove a program and its data) makes all the difference.

Navigon was hanging, and I wanted to stop the thing. I should have taken the top path in the figure above, but it's difficult to get there: 1.) While the misbehaving app is running I have to double-tap the home button get to the running process list, 2.) I have to start another application, 3.) I have to double-tap again to get the running process list again, 4.) I have to hold the icon of the app which I want to kill (because only now it appears in the list) so they all start wobbling and then 5.) tap the cross, and that's it. No problem. Green arrow. Can't go wrong because it only stops the app.

However, the mere complexity of the procedure above is so, that when you know how to do it, you do it cognitively without the brain. It's called routine.

Navigon was hanging, and I wanted to stop it. But at that time I chose the wrong path (which is much simpler than the above. Yes, simpler to get to the dangerous thing than the ok thing). A.) I pressed the home button once, B.) I held the icon until it started wobbling and then, C.) I had to answer something in a query box, either D.1) Yes I want to delete the application and all its data or D.2) No I don't. I pressed Yes. Red arrow: very stupid. Good for us that Navigon wasn't showing us the way to a hospital.

However, my home work should have been to set restrictions, which would have stopped my ability to remove a program! The black crosses simply won't appear no matter how long I hold an icon. See figure below. I had thought this was something that parents did with a kid's iDevice. Now I understand that it's certainly something I should have done to protect myself from my own stupid actions. And from Apple's elegant but dangerous route - even if by design.

For our case it really wasn't so bad, though. I had also installed Navigon on my wife's 3GS, and we could happily drive on. And the hotell we arrived at in Urbino, Italy, had internet access included. Up with AppStore on the iOS, and a new almost 2 GB download. It's perfect that one purchase covers several machines, and that Navigon exists for both machines!

But Apple - the whole of Navigon was probbaly still silently present in the iOS flash-based memory. How about just asking me if I wanted to wake up the just deleted app? I don't know if the iOS file system has a recycle bin internally, but the functionality would have been nice! Or a third choice to removal, yes, cancel or stack (move to some kind of iOS special recycle mechanism?).

I have reported this to http://www.apple.com/feedback/ipad.html

5 - one.com mail squelched on some routers ::

Usually it works surprisingly well to connect iPhone, iPod or iPad to a new router. However at one of the hotels we experienced problems with mail. Browsing with Safari (and Opera) worked fine. But sending and receiving mail with the iOS mail client failed. More detailed: sending always failed, bet reception a very few times seemed to work. We brought one iPad2 (iOS 4.3.3), one iPhone 3G (iOS 4.2.1) and one 3GS (iOS 4.3.1) - but I only experimented on the iPad.

I think one.com and the router in mind together displayed the problem. Not only the Mail client, but even Safari webmail was not able to log in to my one.com account. Opera Mini webmail with one.com logged me in and I could read mail and also send (see below).

We had one ingoing IMAP server connected, and up to three outgoing SPMT servers. Any combination of these did not seem to help.

I don't know if the router had blocked IMAP incoming port 143 or SMTP outgoing port 25 or 587. I tried to test, and all ports looked blocked, even standard browser HTTP port 80. So, I must have used the tool wrongly, or it

The strange thing is that I wasn't able to set up Gmail in iOS Mail either, even if it's more or less automatic and built-in. Another strange thing: why did the lady I asked, sitting with an iPad, have functional mail, in the same hotel?

[1] "Setting up iPhone Mail/iPod Touch" help on one.com: http://www.one.com/en/support/guide/mail/setting-up-iphone-mail. This is not updated with iPad, which is a shame. They don't support home page editing for Mac either, any type - Mac OSX or iOS. Another shame.

6 - Opera Mini browser and webmail ::

The Opera Mini was version 6.0.0.13548. One.com webmail did work to some extent with the particular router, mentioned above. But there are some points, most of them disregarding the fact of the router, I would assume.

Opera Mini could only handle one session "per Opera". Different windows were not able to have different log-ins or sessions. This mixed up the webmail lists, which pointed to the last log-in list. So, if my wife logged in in one window, and I was already logged in in another - when I came back to mine and pressed mail #3, it confused that with entries in her list.
As when using iOS Safari at home (or at the other hotels) webmail sending of file attachments does not work. Both browsers would know perfectly well that this browser is for iOS, so giving me a possibility to browse pictures should have been piece of cake.
Opera Mini hindered me editing the file input line, which proves that they knew it is of little use.
Opera Mini let med pick out the Send text from the Send button, as if that is of any interest.
Opera Mini could not show pictures in mails.

I have another blog about Opera Mini here: 017
.

024 - 16:9 video aspect ratio with width 720 uses height 404 instead of 405

2011-03-08T20:34:00.017+01:00

iPhone movies..

The iPhone 4 produces QuickTime HD movies (.mov) that are 1280 x 720 pixels (16:9 aspect ratio). Observe that this is not the screen aspect ratio, which is 960 x 640 (3:2 or 1.5).

..do not play on all machines

This is very nice. I can play it on my Mac Mini running Mac OS X Snow Leopard and QuickTime Player 10.0. However, the iBook and iMac (lamp) G4 machines running Mac OS X Tiger with QuickTime Player 7.6.4 only plays the sound but shows the odd picture as it moves on.

But QuickTime Player export to "HD 480p" helped

So, I exported from the newest QuickTime Player to a "HD 480p"-type file. This plays beautifully on the older Tiger machines. The movies are 480 x 270 pixels (16:9). The movies were brighter and better looking than corresponding mpeg-2 (.mpg) movies in 640 x 480 (4:3 or 1.33..) shot with a Sony DSC-W100. And somewhat smaller in file size, I think.

However, export to "HD 720p" intrigued me

But I wanted to try to see how export to "HD 720p" ran on the Tiger machines. Better, but really not. I could accept that. However, I saw that this movie was 720 x 404 pixels, a little more than 16:9. I used the lazy solution at Evalwave Factors at http://factors.evalwave.com/ to factorize.

First, 16:9 is 1,7777777777.....

720 = 2 x 2 x 2 x 2 x 3 x 3 x 5
404 = 2 x 2 x 101
removing common factors 2 x 2 yields
180:101 = 1,78217821782.. is almost 16:9

How about one more pixel (404+1) on the short side:
720 = 2 x 2 x 2 x 2 x 3 x 3 x 5
405 = 3 x 3 x 3 x 3 x 5
removing common factors 3 x 3 x 5 yields
16:9
bingo!

So, why is 720 x 404 used?

Kyle Gilman's page http://www.kylegilman.net/ opens a door for me: "Keep the height an even number. Odd numbers freak out the H.264 codec. In this case 720×405 is closer to 16:9, but change it to 404 and everyone will be happier."

Some day I'll investigate more about the H.264 codec.

However, I do remember from the time I did Fast Fourier Transforms that even numbers were required (even if there are odd number algorithms available). And hardware multiplies by two by shifting left by one bit. This is much faster, and does not build up any errors.

Screens certainly must show the missing 405th line as all dark! Or the other way around, the iPhone (or, really H.264) throws away a line?

Another example is 568 x 320

My daughter sent me this iPhone movie by mail, directly from the phone.

568 = 2 x 2 x 2 x 71
320 = 2 x 2 x 2 x 2 x 2 x 2 x 5
removing common factors 2 x 2 x 2 yields
71:40 = 1,775, some 0.0277777.. off 16:9

What is the "real" 16:9 numbers here, i.e. how much is lost? Around those values there are lots of prime numbers, which are never even. Moving one up or down by 1 does not get me there.

I certainly need help here!
.

023 - Concurrency: one writer and one reader is bad enough

2011-02-10T16:44:00.009+01:00

Error on page 15 of IEEE Software Jan/Feb 2011

Dear Forest Shull (editor of IEEE Software)

There is an error at that page, in the article “Parallelism on the Desktop” (Guest Editors’ introduction: Victor Pankratius, Wolfram Schulte and Kurt Keutzer [1]). The authors have virtually fallen into the same trap as many programmers working with concurrency or parallelism.

“If the shared resource is mutable, programmers must introduce exclusion mechanisms, such as locks in C++ or synchronized methods in Java, to prevent race conditions (where two or more threads change the resource at the same time and introduce an inconcistent state).”

In order to be correct, the last part may instead read: “where even a single writer and a concurrent single reader may introduce an inconcistent state”.

The principle is called CREW (Concurrent Read Exclusive Write), see http://en.wikipedia.org/wiki/Concurrent_read,_concurrent_write: “Concurrent Read Exclusive Write (CREW) — multiple processors can read a memory cell but only one can write at a time”.

The idea of the concurrency primitives is to avoid this type of situation. In the problem domain, CREW shows how bad it is. In the solution domain, it would be the accesses (read or write) that need protection if there is at most one writer.

I may blog about this error at http://oyvteig.blogspot.com/ any day soon. (Done, with minor edits.)

[1] - Read pdf (4 MB)

(29June11: no response to the contents)
.

022 - C1X and C++0x concurrency in C and C++ Working Drafts

2011-01-09T21:18:00.035+01:00

Updated 15Aug11

Intro

The scope of this post is to learn how the renewed C and C++ standards may help programmers of concurrent systems do their job. I will of course look at this through my CSP-type glasses, with years of occam programming and using CSP/channel-type libraries in ANSI C in embedded systems as my bias. (Post 021- The problems with threads takes me up to this point.)

I assume that C1X should end of with C12 for the year's publication data ("formal adoption of a revised standard by the end of 2011, with a publication date of 2012" (C - The C1X Charter)).
And C++0x I assume means that they are late, but from the wiki-page is looks like it's going to end up with C++11. As of 12 August 2011, the C++0x specification has been approved by the ISO. The final working draft before final ISO approval standard is N3291, dated 5 April 2011. This draft is not publicly available; the most recent working draft available is (N3242) dated 28 February 2011. So, it's probably going to be called C++11.

Here are the main references:

C1X
http://en.wikipedia.org/wiki/C1X
N1548 - 2 December 2010
C++0x
http://en.wikipedia.org/wiki/C++0x
N3242 - 28 February 2011 (last before approved)
ISO C++ Strategic Plan for Multithreading
N1815 - 2005

Analysis

Letters-in-words and words that I will use to search in the documents. This should(?) indirectly direct me to all the new keywords:

"thread"
"multi" (task thread -threading)
"concurren" (t cy tly)
"synchron" (ization, ous, asynchron..)
"communicat" (e ion ing)
"atomic"
" race" (to exclude brace..)
"deadlock"
"process" (no such term, but there are some threads by name "process")
"schedul" (e, ing)
"happens before"
"inter-thread happens before"

I'd like to see if I could fill in these points:

Process (no: thread) model (versus or and object model)?
Synchronization model?
Communication model?
Memory model - why is there so much about this?

And

Will the C concurrency features be a subset of the C++ concurrency features?
The difference between third-party thread by library (as discussed in post 021) and language defined thread by library as defined here. Is it assumed that the compiler knows what a process (or whatever) is? The answer to this is a function of what I would see from the above list
Per definition it will it be A Good Thing to build a CSP library on top of these! How "nice" will such a library be, i.e. how easy to get correct? (Again, see post 021)

First comments

My post 021 starts with quoting "It is widely acknowledged that concurrent programming is difficult" (Lee). Future programmers of C and C++ may learn by expensive experience that it is even "very difficult".

These papers start after the climax in the middle of Lee's paper. Where he goes on in the right direction, these papers don't go back to square one. They elevate.

Ok, they add concurrency where there was none before. That is their motivation, and it is an important motivation. But whose idea was it to build this on C and C++? Did they ask?

These people are specialists. The papers are not for amateurs, like I would be in their eyes. I have been hidden from most of the problems they solve during my 30 years of concurrent programming. Starting off with a runtime system in 1980 that we bought from a university person who said his code in PL/M had "no errors". We though he was more crazy than he was! Modula 2 and a runtime system for it. Occam on transputers and occam to C (SPoC). Occam will stay the programmer's love of my life. And then back a few hops to C and CSP-type runtime systems.

I really don't understand which planet the open-std.org is on when it comes to concurrency. In post 021 Per Brinch Hansen is quoted for "that Java ignores the last twenty-five years of research in parallel programming languages.". How many years will future C and C++ programmers have lost?

I am not able to see the answer to any of the questions above. There may be answers, but they seem hidden for me. Except: yes - C is a subset of C++ still, at least when it comes to the memory model. I think.

A language isn't for the future just because "futures" are implemented. This is a pattern that I could easily code in occam and install some 20 years ago. But it's nice to have it in a language. And it is nice to have atomicity described. Etc.

But it is the complexity explosion of the standard that worries me. Inspired by Lee, they are two vast seas of nondeterminism - used in another domain - the definition-of-standard domain.

This probably is very reasonable (from C++ page 1152):

29.6.5 Requirements for operations on atomic types [atomics.types.operations.req]

6 #define ATOMIC_VAR_INIT(value) see below

Remarks: The macro expands to a token sequence suitable for constant initialization an atomic variable of static storage duration of a type that is initialization-compatible with value. [ Note: This operation may need to initialize locks. — end note ] Concurrent access to the variable being initialized, even via an atomic operation, constitutes a data race.[ Example: atomic v = ATOMIC_VAR_INIT(5); — end example ]

Please help me. And if I am able to understand, then I need help with some more.

But this is rather ok (C++ page 11):

1.10 Multi-threaded executions and data races [intro.multithread]

1 Every thread in a program can potentially access every object and function in a program (10)

(10) An object with automatic or thread storage duration (3.7) is associated with one specific thread, and can be accessed by a different thread only indirectly through a pointer or reference (3.9.2).

Let's say that I want to implement a CSP library, and when the second process is on a channel, I want to do the rendezvous memcpy from inside the sender process to inside the receiver process. So now I know that I may code this. In C or C++. But observe the level. It's underneath where I should be.

But isn't that what open-std.org has set out to do for me? Make it possible to build any concurrency paradigm on top of what they now give me? Probably they haven't missed that goal. Stroustrup told me exactly this when I approached him at Simula 67's 25th anniversary in Oslo in 1992: C++ does not have any concurrency to let anybody build their own. Now they are building their own.

Last comments

Adam Sampson's thesis Process-Oriented Patterns for Concurrent Software Engineering that he posted in a comment to post 021 certainly points to another direction of research - and present good practices. And I must repeat: the Go programming language faq also referenced in post 021.

Postscript

Is the world really crying for these two houses built on sand?

What did you say, do you need a C/C++ concurrency expert? If you find a good one, you'd need to wait 5-10 years before you know whether he was as good. And he wouldn't know if the house burnt. Working for other companies, making more good money.
.
.

021 - The problems with threads

2010-12-29T12:18:00.062+01:00

Updated 8Jan11

Intro

Being year's passover 2010/11 it may be late to comment on a paper I found in a pile of IEEE Computer magazines. The article dates back to 2006. The article is available on the net as a report from the University of California at Berkeley:

The Problem with Threads
Edward A. Lee
Electrical Engineering and Computer Sciences
University of California at Berkeley
Technical Report No. UCB/EECS-2006-1
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.html (pdf)
January 10, 2006

IEEE Computer 39(5):33-42, May 2006

That IEEE Computer magazine was themed "Future Operating Systems (and) Programming Paradigms" and the article was a cover article.

I advice you to read the well-written article. And it's outspoken and fun reading!

The quotes I use here are from the report (IEEE Computer have edited a little).

Disclaimer

This note's title uses plural "problems", not to make the titles equal - and it is a mix between a review and my own personal views and experience.

Over some years I have published along the line of that article, see http://www.teigfam.net/oyvind/pub/pub.html. I also have some posts here about the same theme, see http://oyvteig.blogspot.com/2009/07/013-overview.html.

"It is widely acknowledged that concurrent programming is difficult" (Lee)

The magazine article says "Concurrent programming is difficult,..". This is what caused my interest, in addition to the title. But this first sentence seems to contradict the title. Why doesn't he say "..that programming with threads is difficult"?

I have attended the WoTUG- and now CPA- series of conferences for quite some years. What I have learned over the years is the opposite:

"Contrary to the belief out there, concurrent programming is easy" (WoTUG/CPA)

Here is the semantics of that statement: "Programming with threads is difficult, but programming with processes is easy". Later in his paper, I think this is also what Lee says. But he feels bleak about the future, while the conferences (and Lee's paper) try to do something with it.

When I programmed in occam 2 for some ten years, concurrent programming was indeed easy. I had no interrupts, only processes, so asynchronous I/O was tamed before it could overwrite common data. I had fully encapsulated processes, with parallel usage rules checked by the compiler. Processes only communicated over zero-buffered synchronous unidirectional named channels. Buffering had to be coded explicitly. Functions were side effect free, as they were not allowed to use channels. And, the compiler checked aliasing errors (so linked lists were not possible..). Occam 2 has been developed since it died as a commercial language, and now holds a plethora of new primitives.

Yes, I did have to learn the art of the trade. Some has called it Process Oriented Programming.

"The problem is that we have chosen concurrent abstractions that do not even vaguely
resemble the concurrency of the physical world." (Lee)

Even if "not even vaguely" is Lee's falsification term here I would disagree.

The physical world is one axis, concretizations of abstract descriptions another. If my concurrent abstraction may be given a name so that I understand what it basically does, and may be seen as a cognitive entity, then I'm ok. I think this is what Wittgenstein in 1921 proposed in his Tractacus.

"Channels", "rendezvous", "port", "protocol", "process" and "alternation" are taken from daily vocabulary. These are good terms that I grasp, but they fit more or less to their physical (if any) counterpart. These are "vague" terms, really - so Lee might have his words ok.

The idea that "object" as used in "object orientation" (OO) should resemble physical world objects is not needed for me. It has been used to describe OO ("object car has wheels"), but after those examples then just forget: "window has buttons" or worse "window has drop-down menu" makes little sense for real windows.

By the way: "thread" is perhaps a good term! I have seen the result when a cat has played with a yarn of a long single thread. Maybe they should have used better weaving terms? Weaving could be seen as quite formal! And if the terms chosen had been better, maybe the solution would have been better, recursing into Lee's statement?

Process model

In my opinion the CSP "process model" is much easier to understand than "object model". The definition of an object these days emphasizes encapsulation more than before. Some years ago, inheritance and polymorphism were more "modern" to include in the object model.

"Nondeterminism of threads" (Freely from Lee)

I'll fill in with an example. The libcsp library by R. Beton furnished Posix light weight thread programmers with a clean CSP API. However, the struct CSP_Channel_t contained three pthread_mutex_t and two pthread_cond_t, in addition to a CSP_Alt_t that contained one of each. Reading it out aloud:

one channel at most needed 4 mutexes and 3 condition variables (*)

This, just to attempt to tame the "nondeterminism of threads" as Lee calls it.

(*) I don't know how many of these are there to handle asynchronous I/O (like interrupt) or preemption.

You could argue that's it's the channel construct that's wrong, but then bear in mind that it is possible to build a "naked" CSP library without any mutexes, condition variables or critical regions. Again, asynchronous I/O of course needs to be handled. However, preemption has not been needed in any CSP-type embedded system I have seen, but I have learnt that it is necessary for an experimental language called Toc which handles deadline scheduling, see this paper by Korsgaard and Hendseth.

Back to nondeterminism. I learned something from Lee's description. I have always though of nondeterminsim as what (should) happen in the occam ALT struct (as opposed to PRI ALT), or what happens with Promela's :: operator. These nondeterminisms are wanted, and would explicitly read "if more than one choice are ready, select any one of them". If that behaviour is not what I want, I instead use the determinsitic construct.

But here Lee points out that it's unwanted altogether in the thread'ed world. (Later on he goes on to describe the "judicious nondeterminism" - which is the one I am used to.)

WYSIWYG semantics (Welch)

"In fact, we have to know about all other threads that might execute (something that may not itself be well defined), and we would have to analyze all possible interleavings. We conclude that with threads, there is no useful theory of equivalence."
...
"Threads, on the other hand, are wildly nondeterministic. The job of the programmer is to prune away that nondeterminism. We have, of course, developed tools to assist in the pruning. Semaphores, monitors, and more modern overlays on threads (discussed in the following section) offer the programmer ever more effective pruning. But pruning a wild mass of brambles rarely yields a satisfactory hedge." (Lee)

Welch here has introduced the term WYSIWYG semantics. Lee in effect shows that threading does not have this type semantics.

I have discussed this in note [007 - Synchronous and asynchronous]. Here is a quote from Welch et.al:

"One crucial benefit of CSP is that its thread semantics are compositional (i.e. WYSIWYG), whereas monitor thread semantics are context-sensitive (i.e. non-WYSIWYG and that's why they hurt!). Example: to write and understand one synchronized method in a (Java) class, we need to write and understand all the synchronized methods in that class at the same time -- we can't knock them off one-by-one! This does not scale!! We have a combinatorial explosion of complexity!!!"
(Letter to Edward A. Parrish, The Editor, IEEE Computer. Peter Welch (University of Kent, UK) et al. http://www.csp-consortium.org/images/IEEE_Computer.pdf (1997)

Observe that I in note 007 discuss this also relative to asynchronous "send and forget" message based systems.

"This pattern can be made to work robustly in Java" (Lee)

Lee shows how easy it is to make erroneous multithreaded code in Java. The observer pattern for one thread, and then the obvious thread safe version is shown. Only, the "thread safe" is only an attempt at it, as Lee shows. There is a potential deadlock lurking.

A couple of memories spring to my mind.

First, before the WoTUG-19 conference I thought that it might be a good idea to present the commstime program (a very simple but effective benchmark program for concurrency) in the new Java language. I did, but concluded that the "Java 1.0's PipedInput/OutputStream classes seem impractical to use because of an implementation error at Sun". I had my fingers burnt on trial one.

But here was a language with some built-in mechanisms for concurrency, so the community started building CSP libraries in Java. Commercial occam was dying, and so was the transputer. There was a void. So JCSP (Communicating Sequential Processes for Java) was developed by University of Kent at Canterbury. JCSP is being updated to have become a viable concurrency platform for Java; I note that last public release is from August 2010 (this is written in Dec. 2010).

There were all these experts, learning all the intricacies of synchronized and Java's monitor. By 1998 all seemed to work fine. Until a student some two years and who knows how many uses later found a simple case where it deadlocked. The result was that parts of JCSP was formally verified and then fixed. Read about this exiting adventure here ("Formal Analysis of Concurrent Java Systems" by Peter H. Welch and Jeremy M.R. Martin). The error was subtle, but the fix was simple.

Second, about the same time, Per Brinch Hansen wrote a letter that floated around on the internet, later to have become seminal. It is very interesting reading, because what he's basically saying is that the Java designer's at Sun hadn't done their homework. Here's his ingress:

"The author examines the synchronization features of Java and finds that they are insecure variants of his earliest ideas in parallel programming published in 1972–73. The claim that Java supports monitors is shown to be false. The author concludes that Java ignores the last twenty-five years of research in parallel programming languages."
(Hansen, Per Brinch, "Java’s insecure parallelism" in ACM SIGPLAN Notices, V.34(4) pp.38-45, April 1999. http://brinch-hansen.net/papers/1999b.pdf)

Per Brinch Hansen then goes on to describe exactly what's wrong with Java. Observe that Brinch Hansen in 1993 invented the SuperPascal language, whose concurrency features are based on a subset of occam 2!-)

"Fixing threads by more aggressive pruning" (Lee)

Lee describes how they in the Ptolemy project ("an ongoing project aimed at modeling, simulating, and designing concurrent, real-time, embedded systems") used several established software engineering processes like a new "code maturity rating system", design reviews, code reviews, nightly builds, regression tests, and automated code coverage metrics.

"The strategy was to use Java threads with monitors." (Lee)

I wander if they had had access to Per Brinch Hansen's paper, or the Welch and Martin paper. Surely they must have been familiar with the problems.

Even with all this going on for them, Lee describes that "No problems were observed until the code deadlocked on April 26, 2004, four years later." Lee goes on to say that:

"Regrettably, I have to conclude that testing may never reveal all the problems in nontrivial multithreaded code" (Lee)

When a new version of some software is released, there has to be thorough testing. Case closed. But are we some times testing too much? Relying on testing where relying on other means would have been better?

Lee points out that avoiding deadlocks in a system based on semaphores is just to follow some well-known rules:

"Of course, there are tantalizingly simple rules for avoiding deadlock. For example, always acquire locks in the same order [32]. However, this rule is very difficult to apply in practice because no method signature in any widely used programming language indicates what locks the method acquires. You need to examine the source code of all methods that you call, and all methods that those methods call, in order to confidently invoke a method. Even if we fix this language problem by making locks part of the method signature, this rule makes it extremely difficult to implement symmetric accesses (where interactions can originate from either end). And no such fix gets around the problem that reasoning about mutual exclusion locks is extremely difficult. If programmers cannot understand their code, then the code will not be reliable" (Lee, italics by me)

In other words semaphores don't have WYSIWYG semantics!

Use of software patterns

Lee also discusses patterns. I have myself used patterns like the "The 'knock-come' deadlock free pattern", described in Note 007. But Lee has a point when he states that:

"Programmers will make errors, and there are no scalable techniques for automatically checking compliance of implementations to patterns." (Lee)

This is very true. But the shortest programs that function well with a pattern may become part of a repository for it. Real coding undermines patterns! Lee goes on:

"More importantly, the patterns can be difficult to combine. Their properties are not typically composable, and hence nontrivial programs that require use of more than one pattern are unlikely to be understandable" (Lee)

This is less universally true, I guess. If a pattern is based on composable components with WYSIWYG semantics, then combining these patterns should be ok semantically. Some times it's wanted to change Master and Slave processes in knock-come. Doing this has no other side effect than wanted. In a real-time system we need to know the timing constraints. If any process decides not to listen on a channel for some time, say 100 ms, then any one sending on that channel would have to block 100 ms. This blocking is no worse than an asynchronous message not been handled by some state for 100 ms. In fact it's better: the blocking doesn't fiddle with the toothwheels, since it would not even see a message that it would else have to schedule for itself some time later.

WYSIWYG semantics and timing constraints

This takes me to a point that I am not so sure about. WYSIWYG semantics will not include timing constraints? In such a system the protocol between processes describe all I need to know? Right? No, I don't think so. I think I need to describe (as a comment or whatever) what the maximum blocking time is. Or really, maximum time between any send and response (in any message based system).

Usually one just discards this problem by stating that an event-action takes zero time. For most systems this is ok. But if we may block 100 ms, this needs to be propagated, just like with the Toc language mentioned above by Korsgaard and Hendseth.

If a protocol is decorated with timing constraints and those are propagated, then You Can See What You Get. But only then.

Time messes up any good argument!

Real-time Java

Rounding the century my company (Navia/Autronica) was a member of the J Consortium - Real-Time Java™ Working Group. It was set up by NIST and backed by HP, Microsoft, Newmonics and others. These days, the www.j-consortium.com honours its ancestry, but now it's about "real-time medical application".

I was the one doing the work for Navia/Autronica, in the Real-Time Java Working Group (RTJWG).

I remember the endless discussions about how to handle "asynchronous I/O". I had been working occam for 10 years then, and that problem as a problem was non-existing. As mentioned earlier, interrupt handlers were proper occam processes, where the interrupt vector was mapped as a channel, possible to use in an ALT (to be able to receive outputs as well). Results were preferably sent on a channel that never was in a steady state to block (by use of an "overflow buffer" process (two processes in series)). This was so nice, and there was no new thinking about an interrupt process or any process. They were born equal. The process model stayed.

"Alternatives to threads" (Lee)

Lee discusses several alternatives, but for me the most interesting is the fact that the Ptolemy II project also has a "rendezvous domain CSP" API. Here is their class hierarchy. They state that it's based on C.A.R. Hoare's 1978 version of CSP. A process in the 1978 version sends to a named process, whereas in the 1985 version a process sends to a named channel, as in occam. I believe Hoare changed it after the Ada experience, in which his 1978 solution hindered building of precompiled (?) libraries. (Please send me a mail if Ptolemy II really sends on a named channel, so that I can fix.)

Using a CSP class library is A Quite Good Thing. This criticism falls in Ptolemy as well as JCSP (mentioned earlier). If a channel (or whatever well thought out concurrency communication primitive..) is not a "first rate citizen" of the language CSP (or..) will "never" succeed if this is all that's supplied in the future.

In a way it's "pruning of nondeterminstic" cognitive thinking, where each time something is read it has to be understood through a hierarchy. Every time! Not nice to say about OO which has its benefits and uses! A compiler would not know what a process is, it can't do parallel usage checks. At best it would be an add-on tool.

"Coordination languages" (Lee)

Finally Lee taks about coordination language, which would sit on top of a more a standard language (with new keywords?) The Ptolemy II rendezvous CSP "framework is properly viewed as a coordination language". There nondeterminsm is explicit when wanted, else it's determinstic.

After Lee's 2006 article: "The Go programming language" (Google)

But A Good Thing, invented after Lee's paper, is Google's Go programming language. Its concurrency is based on the CSP model, and uses channels as interprocess communication. Even if it "does not provide any built-in notion of safe or verifiable concurrency", it is a step in the right direction. I let the Go people have the final words in this note:

"Why build concurrency on the ideas of CSP?
Concurrency and multi-threaded programming have a reputation for difficulty. We believe the problem is due partly to complex designs such as pthreads and partly to overemphasis on low-level details such as mutexes, condition variables, and even memory barriers. Higher-level interfaces enable much simpler code, even if there are still mutexes and such under the covers.
One of the most successful models for providing high-level linguistic support for concurrency comes from Hoare's Communicating Sequential Processes, or CSP. Occam and Erlang are two well known languages that stem from CSP. Go's concurrency primitives derive from a different part of the family tree whose main contribution is the powerful notion of channels as first class objects."
(Go programming language faq)

End
.
.

020 - Mac OS X network disk partially seen

2010-09-19T21:36:00.043+02:00

This Network disk is connected via USB on an AirPort Extreme (7.4.2 and also the better 7.4.1 (*)). Mac OS X 10.6.4 Snow Leopard (but I have also noticed this on earlier versions). (Not fixed with "Apple Addresses AFP Vulnerability With Security Update 2010-006" in Sept. 2010). Tiger 10.4.11 on iBook G4. Disk connected to AirPort Extreme is via USB, and its a Western Digital WD My Book 1028, formatted as "Mac OS Extended (Journaled)".

(*)

New: follow on 004.6 - Airport Extreme version juggling
I am following this situation after the AirPort Extreme update from 7.4.1 to 7.5.2 in Dec. 2010. I wonder why Apple did such a version quantun leap.
I am also following the situation after the update of the Snow Leopard version of the AirPort Tool to 5.5.2 in Dec. 2010. This is the client sw that lets me configure the AirPort Extreme. I do not think that an update of it would have any effect on the situation described here, unless by a secondary effect coming from any error imposed by the AirPort Tool doing wrong configuring of the AirPort Extreme and its file server. (On Tiger 10.4.11 the AirPort Tool is 5.4.2, opening for configuration clash on the Airport Extreme. Maybe I should only use the newest version? How does Apple avoid any configuration clash?). Reported to http://www.apple.com/feedback/airportextreme.html

Symptoms

6Jan11 New: the disk may have been in need of repair. See 004.5 - Time Machine backup on Airport Extreme's USB connected disk

The Network disk was visible and "mountable" in Finder. Disk and contents. Fine!
But when I tried "Save as" from any application I could see the network disk, but not the files and directories
However, during the same "Save as" I could make a New directory and save the file in it, even if I could not see the rest
Running Spotlight on the Network disk showed me the hits, but double-clicking failed to open it, but gave an error message about "broken alias"
At the same Spotlight result window, the file path at the bottom was not shown. I guess that since the alias was wrong, it could not build any path either

Fix (permanent?)

I tried to find a fix on the net, but failed. However, I reasoned that there must be at least two ways to look at the disk, and the system seems to pick the wrong some times. So I tried this (Disclaimer, I have Mac OS X in Norwegian, so the English texts could be wrong):

Open Finder
Go, Connect to server
There were two previous servers in the "Last used servers" list
I copy-pasted both server addresses into a .txt-file, for "backup"
I tried first the one and then the other
The one that removed all the symptoms above I noted
Then I "Emptied" the list of last used servers (command in Finder:Go:Connect to server:look at the list)
Finally I copy-pasted back the one that worked
Now the system has one to look for, and since it's the only reasonable and correct - fine!
I think I have found out that using Go, Connect to server and then use the smb (even if that is the only alternative) seems to work (always?)

It was the "afp://...." that caused my problems, and the "smb://...." that was correct. Please tell me why!

Reported to http://www.apple.com/feedback/macosx.html

Postscript

When I tried this again the other day, there were still two alternatives. The smb is the (right, top) icon with the three persons, which always works - and the AirPort-type (right, bottom) icon is the afp, which does not work. It's all cabled, the AirPort is not on. When I tried a "Save as", it knew that the afb didn't work - see, it was "grayed out", compare with the larger non grayed out alternatives. (I have garbled my disk names). Please teach me! At least, I now know which to use and which not to!

I can see that the Mac OS X displays boths icons for the same disk! The one in the window and the one in the title in the window are one of each! I am lost. But point 10 above is my rope now.

(8April11) See further down (search for "smb") to find a point about a synchronization problem.

My iTunes library and Time Machine on this network disk

After I had changed to smb (above), then iTunes it could not find the iTunes library. It's on a network disk. To recitfy I pointed it to to the smb disk with the three-person icon. All fine, also after I stopped iTunes and started it again. Same for Time Machine!

Another observation with iTunes. It also seems to use several mount paths to the network disk. I can start iTunes, and it shows almost everything, but it displays an exclamation mark in front of the music. Then, the first time I want to play it, it opens a shared network disk dialogue box. When I type the password, music plays. I doubt that iTunes has temporary files on the machine's disk, because it complains immediately if it can't find any network disk.

Tiger - no problem!

Observe that I have no problem whatsoever accessing the same network disk from Mac OS X 10.4.7 Tiger. It seems to always pick afp AppleShare protocol, and the icon is always a globe.

Another problem - a directory's contents invisible on Snow Leopard and not on Tiger

I noticed that synching my pictures on the iPhone from a Mac OS X Tiger machine (iTunes 9.x) all of a sudden showed a picture folder on the iPhone (3G, iOS4.1) that was'n on the network disk that I was synching against. It was called ".... orig" or something along that line. They both showed the exact 100 pictures in each folder, one real and one ghost.

I could see the original network disk directory from the Tiger machine, with contents.

Then I moved synching to the Snow Leopard machine, with iTunes 10. I had to point it to the picture folder of folders on the network disk. After synching the particular folder had gone on the iPhone!

I deleted the iTunes picture data base, and thousands of pictures were rebuilt - but that particular folder was still invisible, both from the Snow Leopard machine and iPhone.

I noticed that both the smb and afb mounts from above showed me the picture folder on the network disk - but it was always empty. I checked access rights all over the place but couldn't find anything. I could not move the directory around, with error (-43) because it could not find the files.

But from Tiger I could move it! (Remember all the other folders were visible on both machines, any mount). But I could not delete anything on Tiger, got error (-50). Tried to restart the disk server on the Airport Extreme. Nothing helped.

Until I made a copy of the folder from Tiger. That copy was fully visible also on Snow Leopard. Then I deleted the original from Tiger, which I now was allowed to! Then I renamed the copy back to the original name.

Then all was visible, and I synched the iPhone from Snow Leopard - and the folder appeared again!

HW connected: see top of note.

Also reported to http://www.apple.com/feedback/macosx.html

Related problem - iPhone synching picture fails on 3G, not 3GS

I just synched my wife's and my own iPhone. She has a 3GS (iOS 4.2.1) and I have a 3G (also iOS 4.2.1). We have separate picture folders on a network disk, connected to an AirPort Extreme (with 7.4.1, see blog 004, post 5 and 6).

I had created two directories from the iBook running Tiger, one on her sector and one one mine. I synched from a Mac Mini running Snow Leopard. I used iTunes 10.2. All this was done on 6Mar11 with all sw up to date.

The picture folders were visible on the 3GS. But not on mine, even after an iPhone restart.

I tried an "old trick" by now: from the Mini I duplicated the directiory on my "sector", deleted the original and renamed it back to the same name. Now it worked! Now also I have the "47 Feb 11" album!

I noticed that iTunes reported that it deleted pictures and then synched some after my "duplicate, delete and rename" exercise! It could look like iTunes saw the pictures on the 3G, but the 3G didn't?

I shouldn't create new directories on a network disk used by iTunes on another machine!

Also reported to http://www.apple.com/feedback/iphone.html

And referenced at http://oyvteig.blogspot.com/2009/07/014-notes-about-iphone.html#014.16

Update: an experiment that went fine

I accessed the network disk that iTunes uses from all three machines: the two with Tiger (iBook G4 and lamp iMac) and one Snow Leopard (Mac Mini). I made a directory with one file from each machine. The files were also originally created on those machines from Grab. Then I duplicated the three album directories into three unit directiories, one for each unit: iPhone 3G, iPhone 3GS and iPad 2 picture synch directories. After synching - all three album directories were visible on all three units.

This was with iTunes 10.2.1, and it exposed the rather peculiar behavior to have all units' pictures redone (thousands times three). (Update 1: next day it scaled and synchronized all the pictures again! For both iPhones and iPad. Remember, they do have separate picture directories, and nothing has changed from yesterday. Not even the in-logging to the network drive. (6April11)) (Update 2: I think that since I use a network disk, the unneccessary re-synching came because the same disk was mounted both with smb and afp. See further up in this post. When I removed the afb mount, iTunes again only synched what it should and not all.) (Update 3: Was this fixed with iTunes 10.2.2? "Resolves an issue which may cause syncing photos with iPhone, iPad, or iPod touch to take longer than necessary".)

This blurs the previously original point: that the picture directory had to be created from the Snow Leopard machine to be visible on 3G. I observed this again two days ago when I synched with the exact same versions as today. (5April11)

Also reported to http://www.apple.com/feedback/iphone.html (twice)

019 - Play (with) network-disk based iTunes library?

2010-08-21T22:51:00.106+02:00

Updated 5Nov2011.

This post is a follow-up from my post 018 - From concrete CDs to abstract iTunes files. And now the old amp broke. Things have now become somewhat clearer for me. I now seem to know the questions - and some answers!

Fig. 1 - Obsoleted physical architecture. See it here ::

Fig. 2 - Present architecture ::

Before Nov. 2011: http://www.teigfam.net/oyvind/blogspot/019/fig2.jpg
Present architecure

Now, all iTunes files are located on a network disk and the Media center (Mac Mini) is running the iTunes as Master while the other machines run iTunes as Slaves. I have dropped the pseudo multiclient architecture described in 003 - Shared iTunes library on a network disk when iTunes went on to version 10.x but iTunes on Tiger machines now have stopped at version 9.x (Sept. 2010). There is a small description of the new architecture in the blog 003 ref above (blue text there).

The stereo now is a Denon Ceol RCDN-7 with AirPlay, new June 2011. See blog 027 Experiencing Apple AirPlay.

I also see the Apple's plug-in solution with the Remote app works like a dream! The whole library in the hand, with music graphics and all I sure have learned since I, half a year ago, deemed that solution as not practical!

But I had to (or have to?) quarrel with firmware updates to get there! See 004.5 and 004.6.

In Nov. 2011 I gave up on Airport Extreme with a USB connected NAS disk for an Apple Time Capsule with internal disk instead. See 004 - Living with a network disk in the house.

How I got here

The goal is to be able to play our music (and view the CD covers) on all the units above: the Media center, the Laptop and Desktop, the Stereo and the Portable (sitting permanently on the fridge). The existing local net should be used as much as possible. There should be as little direct cabling as possible, like analogue, USB or FireWire or opto-coupled TOSLINK (S/PDIF).

Also, the Stereo should be controllable from the Media center and Laptop. And the Stereo should be able to stream "sound out" from those two units, meaning that we could have better sound while viewing TV or running iTunes on the Laptop - since they are all in the same room.

A goal is to only require the Airport Extreme and Network disk to be constantly running. So, the Media center should not run to be able to listen to the Portable or Stereo. So, I hope not to have to install some server on the Media center to convert from iTunes file reading to datastreams. (Update: iTunes version conflict (above) and JackOSX (below) have already changed this for me!)

The users would be me and my wife. Children are satisfied with the the wifi when they are visiting. And grandchildren are satisfied with what we are satisfied with, for now. If wishes coincide, then one of us must at best yield to earphones, since the iTunes files are readable from several sources, but the iTunes client is not multiclient. However, radio, internet radio, TV, DVD or a book would be acceptable alternatives, as well as the basement shop - or a good walk. Please observe that a "5.1 home cinema system" or a "wireless multiroom music system" is not what we're after! However, we might end up using a component from one of these.

Units in need of replacement

Stereo
Portable
The audio switch

Then existing units, still in good working condition:

Media center computer ("Mini"): Intel Mac Mini with Mac OS X Snow Leopard (10.6.4) (Also controlled from the other machines with a VNC client). This may be replaced later with a newer Mini, one that plays full size AVCHD films
Laptop: G4 iBook with Mac OS X Tiger (10.4.11)
Desktop: G4 «Lamp» iMac with Mac OS X Tiger (10.4.11)
Airport extreme router and disk server: 7.4.1 (7.4.2 is unstable for this usage)
The Screen: is a full LCD TV with tuner taken out of use since the shut-down of analogue airborne tv in Norway in 2009. But it has a HDMI and a DVI input which fit exactly to the Decoder and Mini
TV sound: is terrible as with most LCD tvs, and a pair of Logitech active PC speakers with subwoofer unit are still small enough and good enough for a tenfold enhancement by all measures. But the Stereo is much better, but too old these days.
Other units, as seen in the figure

Other

CDs are already obsoleted, as they have all been imported to iTunes. But they are still their own best backup! However, music is dowloaded more and more from iTunes Store
DVDs are still kept and played on the Media center

Some existing problems

50 Hz hum. The Mac Mini has a TOSLINK opto-coupled and analogue audio output. Since the audio switch does not have a TOSLINK input, I use the analogue connection. Since the switch is also connected to the Screen, then I get hum, barely audible at normal distance. And the switch is not a switch, it's a Logitech volume control box with 3.5 mm aux input, which does not disconnect the other source (TV) when the input jack is inserted. I need a better solution! (This could have been solved with a Logitech Z-5500 Digital 5.1 Speaker System which has a SPDIF/TOSLINK-input - but it's completely overkill)
When I want to play music in iTunes on the Laptop and want some more power, I use a cable from it to the existing Stereo's line input. Surprisingly there is no audible hum. But for a new Stereo I'd need a better solution!
There is no way at the moment to play TV sound on the Stereo. Wiring it is no viable alternative.
Likewise, there is no way to play the Mini's sound on the Stereo

Questions and answers

Yes, I use iTunes for my music handling. I am not afraid, as I believe iTunes and Facebook both lock us in, to count two of themThe Web Is Dead")
Streaming of "my?" music with a Spotify client I seldom do. If all I needed resided "in the cloud", some problems would go and some come - and some stay. The architecture I describe already streams a lot, internally and externally - like internet radio stations. So, when iTunes comes with streaming services, so be it! I would still not delete the music library on the network disk, and still wonder if I ever find time to import all the legacy 33 rpm record albums.
No, I have no plan to switch to Windows, Windows Media Player or Winamp. I know they're good and that people are satisfied with them. Fine!

I am studying http://jackaudio.org/

It supports Mac OS X "Core Audio" (ref), which I learned about in note 018.
It references http://www.jackosx.com/ "Jack OS X - a Jack audio connection kit implementation for Mac OS X. Connecting audio from any OS X application to any OS X application"

I also sent a letter to one of the JackOSX authors, who forwarded it to the Yahoo! jackosx group. See http://tech.groups.yahoo.com/group/jackosx/message/3109. I am now (again) registered as aclassifier there. I certainly hope there will be lots of help! I also am aclassifier at Sonos.

I have now dowloaded JackOSX onto the Media center (Mini).

I now don't have to use the "switch" any more. Hum is gone
The TV sound now comes into the Media center (Mini), and is being routed by JAR (Jack Router), to the line (headphone) output to the Amp (unlike Rogue Amoeba's Airfoil there is no notisable delay, so it's actually useful for tv listening)
Setting up routing with the JackOSX Connection Manager is nice but has to be learned: push one end and then double-click the other end to connect. And stereo means connection '1' and '2' for "left" and "right"
The Mini has to run all the time, since it's not practical to take the Mini out of idle to switch on the TV. I wish I didn't have to. The Mini is very silent, i.e. I can't hear the fan with the present power
The JackRouter does not have a volume control. But the Mac OS X Sound config dialogue box shows that the Line-in and Headphone-out (used as "Line-out") both have. Line-in volume is set in that window only. Headphone-out volume may be set in that dialogue box, in the top Menu line, or with the Remote control. Perhaps this is called the Master or Main volume control? Then, if I connect Line-in to Headphone-out via JackServer (in the JackOSX Connection Manager) - then the Remote control works both for iTunes, DVD and TV! Of course the TV may also be controlled with the Tuner's Remote control, or the Screen's Remote control or Logitech's main control which is a nice physical knob which always works! (See figure below, where I assume that Norwegian should be little problem, Apple's graphics does most by itself)
TV and iTunes could sound simultaneously. It's not enough to switch. One has to turn something on and something off. However, switching the Screen to the Media center disconnects the TV sound, since it comes through the HDMI to the "TV"
Another thing: it's possible to run this sw invisibly. Stopping the JackPilot stops the Connection manager and the Pilot window but not the JackOSX server. Nice!

This is nice but rather complex. Next problem is to find a way to export a data stream to the Stereo and Portable. The JackOSX network module is not in the download since the authors are still(?) working on it. It's got to be more complex. Hmm

Yazsoft is not it

I am looking at the Yazsoft media server http://yazsoft.com (which probably reads CoreAudio from JackOSX server?) and exports UPnP data streams..
..which may be read by a Stereo, being an Arcom Solo unit, see http://www.arcam.co.uk/products,solo,Music-Systems.htm.
The Sonos ZonePlayer ZP120 I cannot see is able to read UPnP datastreams?
The best of the two worlds would be to find something that sees the network disk iTunes library and UPnP datastreams?

Darwin Streaming Server is not it

The Darwin Streaming Server does not seem to solve my needs See http://dss.macosforge.org/. I found this from reading Comparison of streaming media systems at Wikipedia. Here is from macosforge's documentation:

The streaming server supports QuickTime Movie (MOV), MPEG-4 (MP4), and 3GPP (3GP) "hinted" files. Hinting is a post-process that you apply to your movies to make them RTSP-streamable. You can hint them with QuickTime Pro or the hinting tool available in the MPEG4IP package. If you don't hint your .mov's or mp4's they will still be HTTP-downloadable but it will take them some seconds to start playing. You won't need a streaming server for this, just use good old Apache. (http://dss.macosforge.org/post/40/)

"Good old Apache"

Ok, maybe? Entering a url to a datastream plays that datastream. Starting an Apache server is a matter of ticking in a dialogue box. But then, that's not all I want.

Apple AirPlay ::

Apple has already had the AirTunes streaming protocol some time, somewhat discussed in my previous note. I discarded it as pretty uninteresting there.

However, the follow-up protocol AirPlay, introduced by Steve Jobs in Sept. 2010, looks more promising. (Maybe he saw the despair in the notes..).

Apple informs about AirPlay at their page. There's also a wiki-page, of course. Read those now.

Apple say that there will be "featured partners" like "Denon, Marantz, B&W Bower & Wilkins, JBL and iHome" who will produce units that understand the AirPlay protocol. (Most of these companies are "so international" that they present top-level pages pointing to countries, where I found no AirPlay info (7Sept10). But the red should be interesting.)

A company called Frontier Silicon develops solutions for modern radios, like the Pinell Supersound II (blog 18). I would be surprised not to see AirPlay appear from those sources. Not many (any?) present systems will support AirPlay, so I'll have to wait to see it included. It might be worth it. As one of the major vendors answered my question in a closed User group: "Will nn support AirPlay in any of your products?" with the reply "We don't know yet.. with our current line of products it's not possible."

Please see fig.1 and the following (above).

So, why do I think this is for me?

Before I answer, I have a general worry here. See "Rogue Amoeba's Airfoil" (below). It routes AirPlay sound. However, it needs to insert a delay in the sound so that it can be heard in phase on all AirPlay units across the home wired or wireless network. In other words, the tv should also have delayed the video, like Rogue Amoeba's built-in Airfoil Video Player. So, in order to distribute tv, I would have to wait for a tv with builr in AirPlay? So, my general wish to have one audio routing sw only may not be easy to get for a while?

I can play iTunes on any machine and treat my new Stereo (with AirPlay!?) as an external speaker, just like with Airport Express. Nice!
It would work over WiFi and Ethernet. So, the Portable (with AirPlay) could equally be an external speaker!
I would be able to stream also from iPhone to any of these units!

And what would I certainly hope for?

I would hope that I also could take the sound from my tv and pick it up (via JackOSX server?) and route that sound also to the Stereo and Portable. In other words: it would also stream non-iTunes streams. I can't see why this should be a problem, I can already play radio url's in iTunes. (Update 3Jan11: see above, AirPlay probably delays sound like Rogue Amoeba's Airfoil, so this is not really possible?)
I would hope that the Stereo and the Portable, which do not have iTunes (but support AirPlay) would also be able to serve as a client, i.e. I would be able to browse the iTunes library, with artwork and info and all - without running iTunes on any machine. Like how the Sonos ZonePlayer ZP120 would (I have assumed) be able to read my network disk based iTunes library
I would certainly hope that the AirPlay sources would be available under macosforge. That way, any programmer like myself or the JacOSX people could get it going as well! Update: forget it! This most probably needs an AitPlay chip to run.

Posted to http://www.apple.com/feedback/itunesapp.html

Interesting reading is "Forget Apple TV. AirPlay Is Apple's Sneak Attack On Television" at Gizmodo which tells that there's a hw chip needed(?) for AirPlay - and asks if it will coexist with UPnP, DNLA, and Windows 7 streaming. That article points to this BridgeCo blog. Read on!

There also is an interesting post "AirTunes v2 UDP streaming protocol" at http://blog.technologeek.org/airtunes-v2

Rogue Amoeba's Airfoil (for Mac) seems to be AirPlay companion

Conclusion: since I must also run the JackRouter, I don't need Airfoil before we also buy AirPlay hw. And then I would need to run both JackRouter (for tv) and Airfoil (for sound only or video played with the built-in Airfoil Video Player).

It seems like I could "Send any audio from your Mac to AirPort Express units, Apple TVs, iPhones and iPods Touch, and even other Macs and PCs, all in sync!" with Rogue Amoeba's Airfoil for Mac. I will test http://www.rogueamoeba.com/airfoil/mac/. It may even outfunction the JackRouter. (The new version 4 in late 2010 seems more interesting than the version of Aug. 2010 when I first discovered it, partly because my target has moved..)

However, "Airfoil recognizes only AirPlay (formerly AirTunes) devices" (reply to support mail). Therefore Airfoil sw will not see or be seen by Sonos devices, since Sonos doesn't support AirPlay with their present range of hw (as of late 2010).

Rogue Amoeba does not have any white paper to describe Airplay's sw architecture. "As for a white paper, no, sorry" (reply to support mail).

In my architecture TV sound comes in on sound system input. There is no point in taking "live" sound from that input! Simply because Airfoil delays all inputs by some 1-2 seconds. They need to do it like this so that they can play the music at different locations over the wired or wifi local network with all speakers in phase. Even if AirPlay may be designed for real-time response, since they run on top of ip or wireless protocols I don't see how they could ensure unnotisable delay. With regards to the fact that I'd like one sw sound component only, this worries me! Maybe I'll have to run JackRouter for tv sound and Airfoil for iTunes in the future? (This is the reason why they have a separate Airfoil Video Player, so that they can delay the video playback as much as sound playback.)
.
.
.

018 - From concrete CDs to abstract iTunes files. And now the old amp broke

2010-04-24T20:49:00.064+02:00

Latest (Aug.2010): This post is a discussion with myself, and has mostly been closed. Read 019 - Play (with) network-disk based iTunes library? as the next and more mature chapter.

My excuse for keeping this blog somewhat unorganized is that it is being written now, while I try to get organized. I am writing it to help me get organized. And, perhaps you could reckognise your situation?

Intro: how? to play my music

What can help, now that my LP vinyl collection has been stacked away for almost 20 years, and I have reached "9.5 days" of playing with CD import to iTunes, and still have a "day or two" left to import? In other words: I don't have to get more CD shelf space, it's opposite: I need less and less. In practice this also means that the the old Denon CD player DCD-680 will be out of use pretty soon. And it probably won't even be kept as a spare, as has been the old 33 rpm LP player. It was bought when I acquired the CD player, to make it possible to play LPs indefinitely. It still works. However, since CDs can be played (or imported) on any PC or Mac, I am uncertain about the short-term fate of the CD player.

And - for a some time now the old Denon DRA-335R radio and amplifier has been getting rusty in its voice, i.e. broken for any real world listening.

The times they are a'changin, sings Dylan on one of the CDs. Yes, they still are. But how do I cope?

The new backbone: a network disk with iTunes library

My new backbone is not the shelf with plastic, but a network disk with iTunes files, and a good backup scheme. See blogs [1] and [2]. It's available over wifi , but I have also wired all necessary spots. We use the wires most: it's faster, and we swim in less radio waves (set to 10% power).

But I have to move on with the blog. I'll start off with an example, and then perhaps later on try to set up a specification.

Background

Sonos have a great box called ZonePlayer ZP120 [3]. What's in it for me?

When I say "me" I really mean "us": my wife and me, with no children in the house any more, but they come along visiting, some with grandchildren. How should our new "stereo" be like?

First, we're not talking about a home movie theatre, and no 5.1 surround sound. Not in our living room, and not in any other either. We go to the cinemas, watch tv, and we watch internet tv and DVDs using a Mac Mini and the tv as a screen. The central here is a multimedia rack I made a while ago [4].

However, we are talking about iPhones with playlists. They play home made movies and music in their iPod applications. Quite nice. There is a subtle integration here, with the living room. Even the car. Perhaps also with my workshop in the basement. But how should I do this?

Scenario 1: ZonePlayer ZP120?

This unit could easily replace the old amplifier. It's got a nice Class D amplifier [5] that I'd really like to own, and more than enough power. Power output for stereospeakers and a line subwoofer outputs. It connects to our home's wired network, which is nice - and reads files as well as radio stations out there. It also has wifi, but only over a proprietary SonosNet protocol.

There is a remote control for it, but it costs more than half of the unit. However, a free iPhone App also controls it, and that would help. Especially since we have a first generation iPod touch laying around.

Sonos would read iTunes playlists, once it's told by a Mac/Windows application called "Sonos Desktop Controller" where the iTunes library is located (where the file "iTunes Music Library.xml" is located). Even if it's on the home network disk. So, none of my machines need to run while Sonos streams music from the iTunes-loaded network disk. I like it. So, this software does set-up and it also is remote player client or control system for the ZonePlayer. They even say that it shows CD covers, also while playing iTunes [17]. The sw seems to have changed name since I first started this blog. Now (Aug.'10) it's called "Sonos Controller for Mac or PC".

However, it doesn't play DRM and WMA lossless files. So, some of my songs would not be seen. I have not studied the percentage I loose there.

On the other side Apple has this Airport Express that has the feature that it appears as an iTunes output channel [6]. Apple's protocol (AirTunes, in Sept. 2010 replaced by AirPlay, see note 19) is probably not any more standard than the SonosNet. The iTunes would stream to that unit, and I could connect the 3.5mm jack output to the ZonePlayer's input. The Airport Express also functions as a TOSLINK optical digital connector over the 3.5 mm jack [7]. This is beautiful to avoid hum - but the ZonePlayer only has RCA inputs, so I would need an optical PCM to analogue converter to use it [7]. This is a substantial extra cost. Also, a limitation with the Airport Express is that I can only stream from iTunes wirelessly, not via cable. However, the good thing is that Airport Express would be available from any machine that runs iTunes that sees the network disk.

So, how about connecting the output from the Mac Mini (which also has TOSLINK) to the ZonePlayer? With 10 m of analogue wire, I would rather not. I already have a little hum from not using the TOSLINK output to the little Logitec Z4 that replaces the sound from the Samsung tv. When I switch it to the Mac Mini, that's when the hum comes - since I have not used the supplied TOSLINK. The hum is small, but I may hear it on a bad day. But, as mentioned, I could use 10m of optical and an optical PCM to anlogue converter box [7] here. The cost is about 40% of the price of ZonePlayer ZP120.

I am confused. Buy a ZonePlayer just for the amp? Buy another little amp? Use two user interfaces, one for the ZonePlayer and one for iTunes? Drop iTunes? How big is the Sonos lock-in effect. Now that the iPhones already lock us in, I'd not want more!

(Any system would probably try to lock a user in, ref. Gödel and Russell - and the laws of capitalism. So, trying to be outside one could easily be locked out, which is just another kind of lock-in? Standards or lack of standards also cause lock-in.)

I would suggest you also read David Pogue's article in NYTimes [9].

Scenario 2: Just an amplifier: Denon DRA-F107?

This unit costs half of the ZP120. But if I add 10m of optical cable and the optical PCM to analogue converter [7] the only gain is a little lower price and the simplicity. As you understand, there is no TOSLINK input here, no digital audio input. DRA-F107 has a phono input from my still kept record player, to play LPs. It also has an FM radio, and DAB+ may be ordered. It also has a remote controller, somewhat easier than the AppStore remote app. ZP120 has internet radio, but no phono input.

This unit is smaller than the two old Denon 43 cm width units, but a little wider than ZP120.

I want to keep the small speakers we have: Bang & Olufsen Beovox CX50 [8]. I bought new elements in them some years ago, and the play quite well. But I would like a subwoofer in addition, since the CX50 are only 2.5 liter internally.

How easy should it be to play a song?

A. Before

Hmm, I'd like to play a record
Where is it, I think I placed it here. Darn, it's too dark, I can't find the CD.
There it is!
Switch on the units
Push in the CD
Find the song and play!

B.1 iTunes with help from external amplifier (before Post 019)

Hmm, I'd like to play a record
Start the Mac and iTunes (I never switch it off, so I only need to wake it up)
Switch TV to become Mac mini screen
Since I played iTunes on another machine before, I must wait until iTunes library has been updated
Select speakers: the local PC/TV speakers or the nicer across the 10m optical cable
Switch that system on
Play

B.2 iTunes when Post 019 is much easier (all already runs, machine and iTunes):

Hmm, I'd like to play a record
Start Apple's "Remote" app your own iPhone (or iPod)
Or use the remote control that came with the Mac Mini to continue playing. Observe I haven't switched on the Mac Mini's tv screen
Play (it may take some seconds before the network disk spins)

C. iTunes files only with ZP120

Hmm, I'd like to play a record
Switch on Sonos ZP120
Find the iPod and start the remote control App
Have it connect to the ZP120
Wait until the network disk is seen by ZP120
Find the song in the list
Play

I don't know which is fastest, but I suspect A. So, why not keep the old CDs? Because I cannot and want not stop the world alone. I see that with a ZP120 I also have the choice between B and C. But with DRA-107 I only have B.

Longevity of living-room music playing instruments

So, how technical should it be? It is good that a two year old grandchild never would be able to start her favourite music alone (B and C, above). And I, being a computer programmer, in a way don't like B and C. They are not usage elegant! Just technically elegant!

But maybe they are, after all, two steps ahead and only one back. And I won't come to half way by going two ahead and one back, but I'll come to another place. In iTunes I will be able to see the full collection at once, with front covers. I can search and group and play across albums with playlists. I can easily move music to iPhones etc.

But is iTunes here in five or ten years? I do keep two networks disks: one connected and in the house, the other in another house, which carry over and mirror some times per year. I also have a copy on a machine in house. How do I know that my backup is a valid backup? Of the CDs there was no backup: lost is lost. Hmm.

And how is ZP120 updated? For how long?

Any system that needs backup and centralized storage needs an "adminstrator". That's me in this house! But will I survive my wife, and if so, admin myself? We're still only 60 young, but age is volatile. Hmm.

Ok, I need switch: Presonus Firestudio Mobile or Apogee Duet or iMic?

I want to play iTunes directly from the external disk without any machine. And I want the same speakers to play music from any (?) machine playing iTunes, or any sound from any machine! And I want no hum from them. And I don't want to stream the music over WiFi. Seems like I'm starting to understand what I need?

Here are some products that could help. I could place one of these units beside the Sonos player, and any one of them could be of great help. The Presonus (left in the picture) unit takes both FireWire and S/PDIF (TOSLINK) inputs. I don't think it shows up as an external speaker on my Mac, and that I need Presonus sw to be able to stream music out on that FireWire? However, the Apogee Duet will turn up as an external sound unit because it supports Mac OS X Core Audio.

Thinking about excatly the last point. I have a very old now, iMic USB Audio Interface from Griffin Technology. I have enjoyd it for 8 years, sitting next to a Lamp Mac that I love. It connects to USB and has 3.5 mm stereo input and also output. So it also must support Core Audio, and has since Mac OS X 10.3 I think it was! The iMic is still a product!

However, none of the products function as an USB, FireWire or optical switch. And I have to have wires from each machine up to the ZP120. Am I getting closer, or am I just learning?

Another thing: FireWire or USB from a remote machine, and analogue out into ZP120 - I have a feeling I would still get some hum!

I probably either want an optical switch or to stream music from a sound "http driver" directly into ZP120? That would be nice! One url from each machine! I think I will continue my search. Maybe I'm only a driver away! No extra hw!

New knowledge: UPnP - Universal Plug and Play

Maybe the UPnP - Universal Plug and Play [11] (for Mac, for me) comes to the rescue?

A. C. Ryan has a box that seems to read UPnP servers [12]. There is a lot about iTunes and the like at the discussion thread [13], "iTunes streaming". There is pointed to Allegro [14] and Yazsoft [15] Media Servers for Mac.

And I must find out if Sono ZP120 reads UPnP.

More new knowledge: DLNA - Digital Living Network Alliance

This is an alliance that defines and looks after protocol usage(?) [16]. There is a connection between UPnP and DLNA that I'll have to find out about. More later, since "knowledge" in the headings is very exaggerated. I'll have to learn this!

Sonos Controller for Mac or PC

This is s client to control the ZonePlayer (see above). It is not a server that helps me stream music from an other machine, which is what I am discussing at this stage.

Jack Audio Connection Kit

Just for the record for myself, this is very interesting, and I will certainly look into this more in my next note. Here is a quote from [18]:

What is JACK?

Have you ever wanted to take the audio output of one piece of software and send it to another? How about taking the output of that same program and send it to two others, then record the result in the first program? Or maybe you're a programmer who writes real-time audio and music applications and who is looking for a cross-platform API that enables not only device sharing but also inter-application audio routing, and is incredibly easy to learn and use? If so, JACK may be what you've been looking for.

JACK is system for handling real-time, low latency audio (and MIDI). It runs on GNU/Linux, Solaris, FreeBSD, OS X and Windows (and can be ported to other POSIX-conformant platforms). It can connect a number of different applications to an audio device, as well as allowing them to share audio between themselves. Its clients can run in their own processes (ie. as normal applications), or can they can run within the JACK server (ie. as a "plugin"). JACK also has support for distributing audio processing across a network, both fast & reliable LANs as well as slower, less reliable WANs.

Logitech Squeezebox™ Touch

Closing this name throwing, here is the last piece I need for homework for my next note: the Logitech Squeezebox™ Touch. It takes physical connections as well as works as a streaming client. See [19]. There's also the Logitech Squeezebox Radio, which I will need to look more into.

An "on the fridge" type radio: Pinell Supersound II

I'll watch out for this in my next note. It's a Norwegian radio, but it looks interesting. See [20]. It also reads UPnP streams. It turns out that Pinell is using a "turnkey solution" from Frontier Silicon [22] for their units.

Finally

I will try to connect all this in the next note 019 - Play (with) network-disk based iTunes library?

References

[1] - Blog post: 003 - Shared iTunes library on a network disk

[2] - Blog post: 004 - Living with a network disk in the house

[3] - Sonos: http://en.wikipedia.org/wiki/Sonos

[4] - Furniture design an building: Multimedia rack

[5] - Class D amplifier: http://en.wikipedia.org/wiki/Class_d_amplifier

[6] - Apple Airport Express and AirTunes: http://en.wikipedia.org/wiki/AirPort_Express#AirTunes. In Sept. 2010 replaced or expanded into the AirPlay protocol, see note 19

[7] - TOSLINK optical connection: http://en.wikipedia.org/wiki/TOSLINK
PCM decoding from Mac Mini: S/PDIF: http://en.wikipedia.org/wiki/S/PDIF
PCM adaptor: http://en.wikipedia.org/wiki/PCM_adaptor
Box to convert optical S/PDIF or TOSLINK digital audio to L/R analog audio:
Gefen Digital Audio Decoder DD D/A-converter: http://www.gefen.com/kvm/dproduct.jsp?prod_id=5980

[8] - Bang & Olufsen Beovox CX50 http://www.beoworld.org

[9] - David Pogue (The New York Times), "No ‘System,’ but Music Housewide" - http://www.nytimes.com/2009/11/19/technology/personaltech/19pogue.html?_r=3

[10] - Denon DRA-F107: http://www.hifiklubben.no/produkter/stereo/receivere/denon_dra-f107_receiver_sort.htm

[11] - UPnP - Universal Plug and Play: http://en.wikipedia.org/wiki/UPnP

[12] - A.C. Ryan - PlayOn! Media player: http://www.playonhd.com/en/

[13] - Thread "iTunes streaming" - http://www.acryan.com/forums/viewtopic.php?f=98&t=4029&p=41492

[14] - Allegro Media Server for Mac: http://www.allegrosoft.com/ams.html

[15] - Yazsoft Media Server for Mac: http://yazsoft.com/products/playback/

[16] - DLNA - Digital Living Network Alliance: http://en.wikipedia.org/wiki/Digital_Living_Network_Alliance

[17] - Sonos Controller for Mac or PC - http://www.sonos.com/products/controllers/desktopcontroller/default.aspx?rdr=true&LangType=1033

[18] - Jack Audio Connection Kit - http://jackaudio.org/

[19] - Logitech Squeezebox™ Touch - http://www.logitech.com/en-us/speakers-audio/wireless-music-systems/devices/5745

[20] - Pinell Supersound II - http://www.pinell.no

[21] - Comparison of streaming media systems: http://en.wikipedia.org/wiki/Comparison_of_streaming_media_systems

[22] - Frontier Silicon: http://www.frontier-silicon.com - used by "practically all major consumer electronics brands, including Bang & Olufsen, Bose, Bush, Cyrus, Denon, Goodmans, Grundig, Harman/Kardon, Hitachi, JVC, Magic Box, Ministry of Sound, Onkyo, Panasonic, Philips, Pioneer, Pure, Revo, Roberts, Samsung, Sanyo, Sharp, Sony, TEAC and Yamaha".

.

017 - Safari and Opera Mini on iPhone/iPad

2010-04-13T22:18:00.036+02:00

This blog will show how a table intensic page (my home page at http://www.teigfam.net/oyvind/) is rendered on the iPhone by Safari and Opera Mini. Naturally, for 10 years the page has been my private browser "acid" test. Since there are billions of other pages to compare with, I'll stick to this.

Observe that iPhone (original, 3G, 3GS, 4) has a 3/2 screen (480/320) or (960/640) while iPad (1,2) has a 4/3 screen (1024/768). My index file uses table layout to percentage of screen, not pixel counts. Except for icons, which have to be pixel-defined. H, T, P and F are also graphics. Opera tries to use some heuristics for the small screens, which are basically not successful. Safari does what it's told and does not attempt to be smart, which is super.

1 - Initial version of Opera Mini

Safari on iPhone 3G 3.1.3 (7E18)
Opera Mini Web browser, v5.0
13 April 2010

Verdict: Presently I will use Opera if I need to search for text in a page, need to read it off-line, or have to download a large page with per-MB cost over my phone connection (good speed and also optionally no pictures). This means that I'll mostly stick to Safari, since it's also an iPhone Application, not a port. One "star" for this version.

1.1 - Safari has always been super

1.2 - Opera Mini on iPhone was bad

2 - iOS 4 with Opera didn't help as Opera was the same

Safari on iPhone 3G 4.0.1 (8A306)
Opera Mini Web browser, v5.0 (5.0.0119802 2010-05-06)
26 July 2010

Safari is new, but the layout is 100% equal.
Also no change in Opera Mini's layout. It's the same version so that's hardly no surprise.

3 - Opera Mini 6 on iPhone is getting better

iPhone 3G, iOS 4.2.1
Opera Mini 6.0.0.13548
22 June 2011 (the file has modified contents, of no importance)

4 - Opera Mini 6 on iPhone 4 oops! and getting better

iPhone 4, iOS 4.3.3
Opera Mini 6.0.0.13548
24 June 2011
Showing two standing pictures here. Same iOne
Left screen cut reported to http://mini.bugs.opera.com/

5 - Opera Mini 6 on iPad is super

iPad2, iOS 4.3.3
Opera Mini 6.0.0.13548
23 June 2011
The pixel real estate available is about the same as on the iPhone 4, yet layout here is super. Do Opera Software use another algorithm, on board iPad or on their layout server?

016 - Cooperative scheduling in ANSI-C and process body software quality metrics

2010-03-17T21:25:00.006+01:00

Background

The software metric STPTH ("static path count") some times yields an unacceptably worrying figure. The higher the worse, like 6000, which is really bad - signifying a high code complexity. However, is it the code or the formula that's unacceptable? Let's try to find out.

STPTH - what is it?

STPTH seems to be defined in the now superseded ISO/IEC 9126 Quality Attribute Model [1]. The [1] Wikipedia article does not mention the sub-characteristic attribute tree: maintainability - complexity - static path count, which [2] does. The figure is a scren clip from [2].

I have not been able to find the formula, but the number of branches is in it. However, in order to clarify, let's look at the difference between STPTH ("static path count") and STCYC ("cyclomatic complexity"). Quoting from a mail, where I had queried about these matters:

..It is very easy to generate code with a large value of STPTH because the algorithm for computing STPTH is simplistic. For example, the addition of a switch construct with 5 branches will add 5 to the value of STCYC but will multiply the value of STPTH by 5.

In other words: if then else and switch case code is expensive STPTH-wise. But watch out, there is at least a case where STPTH is not very meaningful: high number is ok. It's worse: the normal figure is high!

Recently I queried the English company Programming Research and they replied:

This is similar to Nejmeh’s (1988) NPATH statistic and gives an upper bound on the number of possible paths in the control flow of the function.

I found a reference and explanation in [3]. Npath is the name of a tool computing the NPATH measure and other measures. Quoting from [3]:

The NPATH metric computes the number of possible execution paths through a function. It takes into account the nesting of conditional statements and multi-part boolean expressions (e.g., A && B, C || D, etc.). Nejmeh says that his group had an informal NPATH limit of 200 on individual routines; functions that exceeded this value were candidates for further decomposition - or at least a closer look.

Also, [3] references the real source of it all, namely and article by Brian Nejmeh in 1988 [4].

An example

Programming Research gave me this example, knowing that I was writing this blog, so I assume it's ok to quote here:

The following code example has a static path count of 26.

if ( ) { } // block 1, paths 1 else if ( ) { if ( ) { } // block 2, paths 1 else { } // block 3, paths 1 // block 4, paths block2 + block3 = 2 switch ( ) { case 1 : break; case 2 : break; case 3 : break; case 4 : break; default: break; } // block 5, paths = 5 } // block 6, paths block4 * block5 = 10 else { if ( ) { } // block 7, paths 1 else { } // block 8, paths 1 } // block 9, paths block7 + block8 = 2
// block 10, paths block1 + block6 + block9 = 13
if ( ) { } // block 11, paths 1 else { } // block 12, paths 1
// block 13, paths block11 + block12 = 2 // outer block, paths block10 * block13 = 26

Each condition is treated as disjoint. In other words no conclusions are drawn about a condition which is being tested more than once. The true path count through a function may therefore be lower than the estimated static path count but will never be less than the Cyclomatic Complexity (STCYC).

The true path count through a function usually obeys the inequality:
Cyclomatic Complexity <= true path count <= Static Path Count

Our code example

The code is discussed in [5]. Here is the start of it:

There are 3 visible synchronization points in this code: lines 08, 17 and 21. The channel based scheduler is one return away from the code you see here. The PROCTOR_PREFIX in line 04 is an invisible branch table to invisble labels hidden inside the macros of the synchronization points.

When a channel (or one of a set of channels in the ALT) is first on the channel, there is a return to the scheduler, to implement the blocking semantics of a synchronous channel based scheduler. This scheduling is called cooperative scheduling, and all the code you see is plain ANSI-C with zero tricks - other than the fact that the branch proctor table is made with a script which searches for the labels. "The channels is the scheduling" with these systems, except for the asynchronous no-data signal channel we have implemented. The Wikipedia article [6] is not very relevant though, since it does not explain cooperative multitasking in use for a CSP (Communicating Sequential Processes) based system (March '10). Ada could have used that scheme, occam [7] does, as does our system. These systems could be, and ours is, non-preemptive. Therefore no assembler or tricks, since I don't think it is possible to write a preemptive scheduler in ANSI C without relegating to assembly. I cannot find a suitable Wikipedia article for this.

The problem with this code is that it is much more complex than it looks.

Some of the channel macros are in-line code with conditionals. (I don't like macros, but with ANSI-C I tend to love them.) And, when we receive something on a channel, it's not handled inside the ALT structure (as would occam or Ada), but in the nice switch (g_ThisChannelId) case block below. Also, the fold in line 20, which would contain 5 case branches. Inside these cases one would usually check the received protocol in a level or two, and then innermost we could send off a CHAN_OUT, which

This inner CHAN_OUT cannot be done from a function! There is no other way, with the non-preemptive cooperative scheduler we use.

Aside: it's not as it was with the SPoC (Southampton Portable occam Compiler, [8]) which generated ANSI-C from occam. Occam allows communication on channels from any PROC (except from its side-effect free functions). So SPoC had to allow it. It used the same return to the scheduler (that's where we learned this). The trick was to "flatten out" any PROC by not calling it, but starting it as a special type of PROCess, only to push the return down (any number of levels) to one above the scheduler. Alas, this is too complex for us to do by hand-written ANSI-C!

Still I would argue that the code above is as simple as it could be. In a code example at work, the STPTH value started at 13000, but I was able to reduce it to 6000. Then my willingness to fine tune any more came to a halt.

The fact that I took the energy to reduce by 50% showed that STPTH had some value!

A suggestion

Perhaps the tool that calculates the STPTH for us could see these inner synchronization points that cannot be put into called functions and "excuse" them? We could decorate that inner point with an exception rule that could enter into the formula. That way we could get STPTH down to some value that we won't have to explain.

In the reports we get an overview off all exceptions, so it won't be forgotten.

I would certainly like comments on this. How should the exception modify the STPTH formula?

References

[1] - ISO 9126 international standard for the evaluation of software quality http://en.wikipedia.org/wiki/ISO_9126.

[2] - Improving Software Quality through Static Analysis, in http://findbugs.cs.umd.edu/talks/JavaOne2007-TS2007.pdf (Pugh) and http://www.cs.umd.edu/~jfoster/papers/paste07.pdf (Foster, Hicks, Pugh)

[3] - npath - C Source Complexity Measures, see http://www.geonius.com/software/tools/npath.html

[4] - Brian Nejmeh: NPATH: A Measure of Execution Path Complexity and its Applications, in Communications of the ACM, Februrary 1988.

[5] - New ALT for Application Timers and Synchronisation Point Scheduling. Two excerpts from a small channel based scheduler. Øyvind Teig and Per Johan Vannebo, Autronica Fire and Security (AFS) (A UTC Fire and Security company). In Communicating Process Architectures 2009 (WoTUG-32)
Peter Welch, Herman W. Roebbers, Jan F. Broenink, Frederick R.M. Barnes, Carl G. Ritson, Adam T. Sampson, Gardiner S. Stiles and Brian Vinter (Eds.) IOS Press, 2009 (135-144) ISBN 978-1-60750-065-0 © 2009 The authors and IOS Press. All rights reserved. See http://www.teigfam.net/oyvind/pub/pub_details.html#NewALT

[6] - Cooperative multitasking - http://en.wikipedia.org/wiki/Cooperative_Scheduling#Cooperative_multitasking.2Ftime-sharing

[7] - Occam - http://en.wikipedia.org/wiki/Occam_(programming_language)

[8] - Southampton's Portable Occam Compiler (SPOC) (1994) by Mark Debbage, Mark Hill, Sean Wyke, Denis Nicole. Just search for this on the net. I also have several hands on papers with it, see http://www.teigfam.net/oyvind/pub/pub.html
.
.

015 - In search of Go! programming language rendezvous

2010-01-22T22:47:00.009+01:00

Intro

I would not have known about the Go! programming language [1] if I had not been made aware of the Go programming language [2]. Since I liked so much of what I read about Go's concurrency solutions, what was more close than to have a look at Go!!

(Why Go! with an exclamation mark, a punctuation mark in the name? It's a smart language, but the whole idea of including ! in it, is this the reason why even Google didn't detect? See the mismatch above, the paragraph ended with two !!, not at same semantic level!)

Name aside. I ordered the "Let's Go!" book [3] to have right here, beside me, since originally, I concluded there's no Go book to hold on to yet (Jan.10). The stuff I write here will be with reference to Let's Go! - but there's a "Go! reference manual" that's included in Go!'s download package. You'll find much of it there.

I will do no programming here.

Go! & concurrency

Go! is an object oriented, functional, procedural and logic programming language. I should study it, I see that I have a lot to learn. However, it is also multithreaded. My home field, even if I only know this little corner.

A user "spawns off computations as separate threads or tasks". It handles synchronized sharing of resources and coordination of activities. But a process (I prefer that here for thread or task) is not a primary citizen of the language. If you don't use a shared resource correctly, or don't comunicate between processes correctly, well - it's your problem.

However, there's sync that's built into the language (like synchronized in Java, I guess). It'd better be, it's harder not to, if you need them. Message communication is through a library: go.mbox is the standard message communication library.

Go! & sync

There is a sync mechanism. Any stateful object may be protected with sync. A stateful object uses @> where a statefree uses @= as class constructor. A stateful object seems to be a blacker box, since instance variables are only accessable through access methods. However, there must be more to it than this, i.e. some stateful objects may not be protectable by sync, or some protectable with sync are not stateful (I don't know), since the book points out that deciding whether sync may be used around an object may have to be delayed until run time.

sync may have timeout.

sync may be guarded. According to the value a guard or guards, the sync will pick a choice. I don't know if the language requires the guard to be thread safe, local and not exportable from the process where the guard is used. I'm afraid not. The choice is prioritized, it runs the first (in time) first if several are ready. I don't see any non-determinstic choice. Also, I assume that fairness must be explicitly programmed, if it's a matter of concern. And since the guards only must be evaluated to true, then non-determinism may be programmed through random numbers. (But on a formal level non-deterministic and random choice are not the same [4].)

sync may also be used for notification. So, Java's synchronized and notify is not needed, since it's all in one mechanism here? An object sync'ed on a list may be notified when the list becomes non-empty. Nice! But only one object, so Java's notifyAll is not supported. (But ..All in Java may be too many, so they will have to go to sleep again if it weren't for me..)

In [3] page 177 McCabe points out that

The central message about the use of sync is that it makes for an excellenet way of resolving access contention for shared resources. On the other hand, sync is a perfectly terrible technique for coordinating multiple threads of activity. The reason is that there is no direct way of esatblishing a rendezvous of two or more threads with sync. For that we recommend looking at the go.mbox message communication library.

So, sync makes it possible to make thread-safe libraries.

Go! & Mailbox / Dropbox

First, what is a rendezvous? According to the Wikipedia disambiguation page it's "A communication method of Ada (programming language)". Going into the Ada page there's no mentioned of it (yet..). However, it is a place in code where the first one to arrive (be it sender or receiver) waits for the second to arrive. Then they exchange their data. Then both processes continue. The important things here are:

A rendezvous involves no message buffers
(References to process local data are carried into the rendezvous)
A rendezvous involves blocking
(But it's never busy-polled, and beware that this does not mean that less gets done! At the end of the day in any paradigm, when something isn't ready it isn't.)

I have discussed much of this in my blog 007 - and what it might be good for.

Ada's concurrency is based on C.A.R. Hoare's process algebra CSP (Communicating Sequential Processes). Just like Go's is! (Beware no exclamtion mark). Just like the first runnable language: occam. Just like like the formal language Promela. I say "based on", menaing they are all dialects.

So, I would need to investigate Go!'s mailbox. The concept is that a mailbox is shared, and that processes drop mail in a mailbox with a dropbox. There is one way to drop: use the post method. It is typed of course, but there is no way to block on it. So, the programmer who thinks that things get easier that way (sometimes it does!), just sends and forgets.

A Go! mailbox is for multiple writers, single reader (many to one). It handles coordination, but has specificly been designed not to handle synchronization. In CSP, those are the same things, since scheduling is driven by the rendezvous (or, in the speak I am used to: the channels drive the scheduling, or: the scheduler is just a channel handler (if it's not preemptive, which the model is neutral to)).

Since a Go! mailbox is for multiple writers, single reader, and since there is no addressing or placement, it is the use of them that governs the "addressing". In CSP, one communicates on a named channel (changed from the 1978 to the 1985 version, to better facilitate libraries), which could be not connected or connected or being sent on a channel (occam-pi and Go)). Also, the usage of them is not verified (onlike occam / occam-pi).

The Go! mailbox reader process may block. Blocking means that you don't have to busy-poll for data. A reader may instruct the mailbox to search in the mailbox for matching messages, and block if there isn't any.

This is nice! However, I fail to see the rendezvous in it.

Go! & perhaps some suggestions

I don't know Go! well, and am not aware of the language's application domain, but provided, as the book mentions, a rendezvous with the properties listed above are of any interest to its users, then I have some suggestion.

Observe that I don't want Go! to be equal to any other language!

The suggestion is to add more "symmetry" to Go! The occam realization of has input ALTs with guards, but not output ALTs. Promela has both, with guards. I believe that CSP describes both.

Go! has protected regions with sync with guards, this should be symmetric enough? But it has blocking on read of a mailbox, not blocking on send.

1. Blocking on sending: If I drop "any data" in a dropbox, it could block if the receiver does not need "any data". Or, if the receiver waits for a list of names, then block until somebody supplies it. This could make up an interesting type of many-to-one "channel".

2. Return to known: occam has a way to directly identify where a reply is to be sent. In Go! it has to be coded. Occam does this by putting a many to one channel into an array of channels, and the receiver ALTs on that array. The index of the array falls out of the ALTing even if it's never sent by the sender, so the receiver knows where to send a reply.

3. Propagate guards into the mailbox system: I don't see in Go! that we could put an input guard into the mailbox. If a process wants to accept inputs from set A=[a,b,c,e,f], then if set B=[d] tries to dropbox, it would block. One event in A must happen before B (on perhaps a next round) would be triggered. Guards are constant when they are used.

Would this CSP stuff be interesting to put into an object oriented, functional, procedural and logic programming language with support for concurrency (Go!)? If yes, what would it be good for?

Disclaimer

This note has been written with most knowledge about the CSP type concurrency and some of its implementations. My knowledge about Go! is next to none. Therefore, there may be smart combinations of the concurrency related facets and the other primary citizens of the Go! language that could make this note in error. Please comment and I will change!

Another thing: my suggestions may well fail a verification if modeled in a formal language. Since I have moved things into the mailbox system and included more blocking, it could cause safetly and liveness property violations. In other words, it may not be possible to make a safe language with these features. I hope I am wrong!

References

[1] - Go! programming language, see http://en.wikipedia.org/wiki/Go!_(programming_language)

[2] - Go programming language, see http://en.wikipedia.org/wiki/Go_(programming_language)

[3] - Let's Go! Francis (Frank) G. McCabe (ISBN 0-9754449-1-3), 2007

[4] - Non-deterministic and random choice, a little about this at http://en.wikipedia.org/wiki/Promela#Case_Selection

007 - Synchronous and asynchronous

2010-01-17T13:05:00.040+01:00

Original posting

Intro and quotes

Observe that in 2010 the "action" goes on at the comment and bottom level of this post!

[1] caused me to write this post because I stalled at some of the paragraphs. The authors hit a chord in my favourite mantra - a computer programmer should be fluent at both asynchronous and synchronous systems, and know the pros and cons, necessities and unnecessities of both. I guess that I in this post will try to outline what systems means in this context.

I will start with the quotes from [1]:

[1.1] - "The flexibility required of this missions could not have been accomplished in real time without an asynchronous, multiprogramming operating system where higher priority processes interrupt lower priority processes."

[1.2] - "To our surprise, changing from a synchronous OS used in unmanned missions to an asynchronous OS in manned missions supported asynchronous development of the flight software as well."

[1.3] - "Lessons learned from this effort continue today: Systems are asynchronous, distributed, and event-driven in nature, and this should be reflected inherently in the language used to define them and the tools used to build them."

[1.4] - "Async is an example of a real-time, distributed, communicating FMap structure with both asynchronous and synchronous behavior."

Disclaimer 1: This paper is not a review of [1]. Parts of it was a catalyst to structure some of my own thoughts, many of them are not even mine. I should say again because I have done this over and over. The real theme of [1] I have not even mentioned: the Universal Systems Language (missing Wikipedia page (wrong: it has arrived after my original post [wikipedia] and [12])). The structure(?) of this post is such that I have a virtual dialogue with the authors - based on the quotes above.

Disclaimer 2: This is not a scientific paper - it is a post on a blog.

[1.1] - Synchronous vs. asynchronous OS

[1.1] - "The flexibility required of this missions could not have been accomplished in real time without an asynchronous, multiprogramming operating system where higher priority processes interrupt lower priority processes.

I am not certain what the authors mean with asynchronous, multiprogramming operating system as opposed to a synchronous OS.

But I will give it a try. I assume that in a synchronous OS, the processes (also called tasks) run in some sort of slot, like some milliseconds each. They have to finish their work in their slots, so that the next slot is not delayed. This is not a side effect, it's the whole idea. One process then is not able to interrupt any other - since they run in sequence. Maybe even asynchronous I/O (like polling of interrupt sources) are assigned a slot. With such a system there is no problems with shared access, so semaphores are not needed. Things seem simple.

However, there are several process scheduling algorithms for this [wikipedia]. I assume that some of these would fall into the synchronous OS category. One such scheduling policy could be rate monotonic - where "tasks with shorter periods/deadlines are given higher priorities" [wikipedia].

I feel confident that the authors did not mean that systems - such as those programmed in Ada, with synchronous rendez-vous or channel communication schemes - constitute a synchronous OS at the scheduler level.

Or I could be wrong, and a synchronous OS is the one such that runs and schedules programs of the synchronous programming language type [wikipedia]?

Synchronized OS as using synchronous communication is probably not what they mean - but asynchronous OS and asynchronous communication might be paired in [1]. However, I hope synchronization and OS are orthogonal, also in the paper.

[Def 1] - So I hope that an "asynchronous operating system" could include an Ada-style system without any visible "OS" at all. In the following I have assumed this.

[1.1] again - Required flexibility

[1.1] - "The flexibility required of this missions could not have been accomplished in real time without an asynchronous, multiprogramming operating system where higher priority processes interrupt lower priority processes.

If the authors, with asynchronous, multiprogramming operating system mean that communication mechanism between tasks mostly is based on asynchronous messaging, then I need more explanation than [1] gives me.

However, I think that this is not what they mean. They simply mean that asynchronous OS is not synchronous OS - and that messages (events) between processes drive the system. It's not the passing of a big clockwork that displays time, like the inner machinery of Big Ben in London, where all wheels are synchronous. It's more like the combined effort of any workforce with a loose degree of central control.

The communication mechanism between processes is rather crucial. Some mean that asynchronous communicating (send and forget) and a common message buffer solves their need. Some would say a synchronous communication mechanism solves their need (channels or Ada-like rendez-vous). The latter is often seen as rather stiff or rigid, and I dare to say, little understood. I have written a paper about this: CSP: arriving at the CHANnel island - Industrial practitioner's diary: In search of a new fairway [2].

Using an asynch message scheme would cause a process to have to know how the internal programming in the other processes work. Here is a quote from [3] - which discusses an alternative to the non-Ada type communication scheme, the Java monitor synchronization mechanism:

One crucial benefit of CSP is that its thread semantics are compositional (i.e. WYSIWYG), whereas monitor thread semantics are context-sensitive (i.e. non-WYSIWYG and that's why they hurt!). Example: to write and understand one synchronized method in a (Java) class, we need to write and understand all the synchronized methods in that class at the same time -- we can't knock them off one-by-one! This does not scale!! We have a combinatorial explosion of complexity!!!

CSP is not out of scope here - Ada's rendes-vous is based on CSP [5]. The WYSIWYG is also discussed in [4].

Bear in mind that communication scheme and degree of synchronization are closely related. With zero buffered synchronous channels, communication is synchronization. With buffered asynchronous-until-full channels, synchronization happens when the channel reaches capacity and becomes synchronous. When the channel size is infinite, synchronization never happens - or it does not happen before a sender at application level waits for a response, i.e. inserts synchronization.

Different programmers tend to chose different methodologies here. Or the traditions and earlier usage in different companies. Or changing times. My previous post '006' will try to discuss this.

There are also several other points, like the producer / consumer problem. Here, buffers would fill up (to overflow?) on a faster producer than consumer. Also, with messages flowing into a process, there may be no system to stop a message from arriving. With input guards and synchronous communication it's possible to close the door from a client, to finish off a session with another client - undisturbed by the fact that another message is waiting outside the door. An important building block.

However, since the world is basically asynchronous, it is important that programmers know the different tools and methodologies. Synchronous communication may be built on top of an asynchronous layer - and opposite. I will come back to this. Deadlock (synchronous) and pathological buffer overflow (asynchronous) must be avoided. This is exciting!

[1.2] - Loosely coupled development teams

[1.2] - "To our surprise, changing from a synchronous OS used in unmanned missions to an asynchronous OS in manned missions supported asynchronous development of the flight software as well."

If one of the systems I have been working on is in fact an asynchronous OS, but with synchronous communication (channels) layered on top - and that is the same as the above (assuming [Def 1]), we probably have experienced the same. I have written a paper about this: High Cohesion and Low Coupling: the Office Mapping Factor [6].

I argue that the higher cohesion in each process (they each do a well defined job), and the less coupling (with synchronous communication and WYSIWYG semantics) - the easier it seemed to be for programmers to agree on the interface contract (the message sequences or protocol) - and then enter the office and do the job. There was little need to discuss any more.

My paper is a case observation from industry - so the Office Mapping Factor is only a hypothesis until somebody takes the token and carries on.

In [1.2] there is a term, asynchronous development, which needs a definition. I think they talk about decoupling of the development members and teams as much as possible from each other. My gut feeling is that this is the same as my Office Mapping Factor. However, if they base their software on asynchronously communicating processes - then, they would perhaps get even better yield by using [Def 1] - Ada-type communication.

If the Ada designers had used named rendez-vous or named channels to communicate between processes, instead of named endpoints, it would have been even easier to build good libraries. C.A.R. Hoare, in his second CSP book of 1985 (ref. in [5]) took the channels to be named entities that could be ripped up and placed anywhere - in run-time. A process communicates on a named channel (like on an ip-address), not with another named process. This was about the same time that that Hoare (with fellows at Oxford University) together with people at Inmos (David May etc.) had built a runnable instance of CSP, the programming language occam [7] and a processor for it, called a transputer. It would be wrong of me not to admit that it is from those sources and fellow admirers I have been drinking, from about 1980 to this very day.

This would make the programmer's interfaces (API) better, and well defined message sequences with WYSIWYG semantics - tools for getting "asynchronous development" with high Office Mapping Factor.

[1.3] - Systems are increasingly asynchronous

[1.3] - "Lessons learned from this effort continue today: Systems are asynchronous, distributed, and event-driven in nature, and this should be reflected inherently in the language used to define them and the tools used to build them."

Now I really begin to doubt that [Def 1] is anything but a misinterpretation in the context of [1]. Because, yes, there is a trend. I saw a web page of a high profile lecturer who should have a one day course. One of the points was Running tasks in isolation and communicate via async messages. Then I queried on a discussion group, and one in the community said that:

"Most likely it's the fire and forget and hope the buffer doesn't overflow variety - Erlang style concurrency. This seems to be the current method everyone is interested in (Erlang, Scala actors, F# Async Mailboxes, Microsoft Concurrency and Coordination Runtime). There is sometimes some form of guard available, be it one that allows checking of the incoming message, or selection of one from multiple inputs."

Maybe this is what the authors refer to. Or something along that line.

When they talk about language, I am sure they mean the language described in [1], USL. With solid semantics, and if the semantics is well enough defined, it does not matter if it's based on synchronous or asynchronous communication? Like, Harel State Charts in UML are based on asynchronous communication between state machines. And it's pretty solid, and it doesn't really void formal verification, it seems. Of course it is possible to build good systems with this, both unmanned and manned missions of any literal kind.

But maybe, if the same designers knew as well about the [Def 1] style systems as they do with the asynch OS / asynch communication schemes, we would get even better systems? To introduce a world where it's understood that you cannot simulate or visualize a system to be error free. Some tools now do verification: IAR's Visual State (of UML state machines), Spin (of Promela models) and FDR2 (of CSP), as well as LTSA (of FSP) analysis, see [8]- [11].

[1.4] - Joining heads for synch and asynch?

[1.4] - "Async is an example of a real-time, distributed, communicating FMap structure with both asynchronous and synchronous behavior."

Now, what does this mean? If it's asynchronous OS and synchronous OS behavior, it would sound strange to me. According to Turing, any machine can run on any other. Laid out here: asynch or synch OS may be built on each other. So, the behaviour should really be the same? If MS Word runs on Mac OS X or Windows the behaviour is the same? (well, almost..)

Instead, I have a hypothesis that they this time talk about asynchronous and synchronous communication behaviour? Then it makes more sense to me to talk about different behaviour.

Any software engineer should know these methodologies (tools), know the strengths and weaknesses of each. When it's appropriate to use one instead of the other, and how combinations could be used.

I will not sum up here. I have glimpsed through some points, seriously tearing and wearing on the [1] paper, and used it as a catalyst. I have written about these things for years. Being incomplete as it is, there may be some points in some of my papers, at http://www.teigfam.net/oyvind/pub/pub.html. Maybe I will try to sum up in a future posting.

References

[1] - Universal Systems Language: Lessons Learned from Apollo. Margaret H. Hamilton and William R. Hackler, Hamilton Technologies, Inc. In IEEE Computer, Dec. 2008 pp. 34-43 ("USL"). The authors have commented on this blog, see [C.1] (below).

[2] - CSP: arriving at the CHANnel island - Industrial practitioner's diary: In search of a new fairway. Øyvind Teig, Navia Maritime AS, division Autronica. In "Communicating Process Architectures", P.H. Welch and A.W.P. Bakkers (Eds.), IOS Press, NL, 2000, Pages 251-262, ISBN 1 58603 077 9. CPA 2000 conference (In the series: "WoTUG-23"), Communicating Process Architectures, University of Kent at Canterbury, UK, 10-13. Sept. 2000. Read at my home page at http://www.teigfam.net/oyvind/pub/pub_details.html#CSP:arriving_at_the_CHANnel_island

[3] - Letter to Edward A. Parrish, The Editor, IEEE Computer. Peter Welch (University of Kent, UK) et al. dead url: http://www.cs.bris.ac.uk/~alan/Java/ieeelet.html (1997), internet archive at http://web.archive.org/web/19991013044050/http://www.cs.bris.ac.uk/~alan/Java/ieeelet.html (1999)

[4] - A CSP Model for Java Threads (and Vice-Versa). Peter Welch. Jeremy M. R. Martin. Logic and Semantics Seminar (CU Computer Laboratory) (2000) - http://www.cs.kent.ac.uk/projects/ofa/jcsp/csp-java-model-6up.pdf

[5]- CSP - Communicating sequential processes - http://en.wikipedia.org/wiki/Communicating_sequential_processes

[6] - High Cohesion and Low Coupling: the Office Mapping Factor. Øyvind Teig, Autronica Fire and Security (A UTC Fire and Security company). In Communicating Process Architectures 2007. Alistair McEwan, Steve Schneider, Wilson Ifill and Peter Welch (Eds.). IOS Press, 2007 (pages 313-322). ISBN 978-1-58603-767-3. © 2007 The authors and IOS Press. Read at http://www.teigfam.net/oyvind/pub/pub_details.html#TheOfficeMappingFactor

[7] - The occam programming language - http://en.wikipedia.org/wiki/Occam_(programming_language)

[8] - IAR's Visual State (of UML state machines) - http://en.wikipedia.org/wiki/IAR_Systems

[9] - Spin (of Promela models) - http://en.wikipedia.org/wiki/Promela

[10] - FDR2 (of CSP models) - http://www.fsel.com/

[11] - LTSA (of FSP models) - http://www-dse.doc.ic.ac.uk/concurrency/

[12] - Universal Systems Language - http://en.wikipedia.org/wiki/Universal_Systems_Language

New References from the Comments section (below)

[13] - A Comparative Introduction to CSP, CCS and LOTOS, Colin Fidge, Software Verification Research Centre Department of Computer Science, The University of Queensland, Queensland 4072, Australia, January 1994 - http://ls14-www.cs.tu-dortmund.de/main/images/vorlesungen/KomSer/uebungen/csp-ccs-lotos-fidge94g.pdf

[14] - Metric spaces as models for real-time concurrency,G.M. Reed and A.W. Roscoe, Oxford University Computing Laboratory - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.1311&rep=rep1&type=pdf

[15] - Concurrent Logic Programming Before ICOT: A Personal Perspective, Steve Gregory, Department of Computer Science, University of Bristol, August 15, 2007 - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.88.3100&rep=rep1&type=pdf

[16] - http://oyvteig.blogspot.com/2009/03/009-knock-come-deadlock-free-pattern.html - "009 - The "knock-come" deadlock free pattern" blog here.

[17] - "3. Asynchroneous services for OpenComRTOS and OpenVE" in Altreonic News Q1 2010-1, http://www.altreonic.com/content/altreonic-news-q1-2010-1

Comments

[C.1] - Comments by the Authors of [1], Dec.2009

This post has been commented by the Authors of [1]. The comment has been approved for public reading. Read at http://www.teigfam.net/oyvind/blogspot/007/ieee_teig_notes1.htm.

Thank you for the comments!

I'd certainly like others to comment in this blog! I will try to follow up myself.

[C.2] - CSP and "true concurrency"

Comment [C.1] states that "Note, that for any of the process algebras (CSP, CCS or LOTOS) "parallelism" is a misnomer (see Receptive Process Theory); none of the process algebras support "true" concurrency, only interleaving, see.." [13] page 12.

Page 46 of [13] reads: "True concurrency semantics. Although conceptually simple the interleaving semantics used by the process algebras mean that the “concurrency” operators are not fundamental and hinder the ability to add time to the languages. True concurrency semantics, in which traces become only partially ordered, have been suggested as a more accurate model of concurrency. There are two principal methods: multi-set traces use linear traces with a set of actions listed at each step; causally ordered transitions maintain pointers denoting causal relationships between events in the traces."

This puzzles me, especially since I am a programmer, and not a computer scientist, and therefore would not know the answer. However, I have been running occam on multi-transputers in the nineties, and those processes of course had true parallelism, and not pseudo concurrency. When placing the same processes on a single machine, I certainly experienced their semantics not to change. Occam had "parallel usage rules", helping me to obey CREW (concurrent read, exclusive write) on say, segments of a shared array.

In [14] (page 12) it says: "The semantics we gave for CSP is by no means the only possible one that is reasonable..". "Also both the parallel operators we gave were true parallel operators, in that the time taken by the two operands was not summed: one might well need time-sliced pseudo-parallel operators in applications." Does this contradict the page 46 quote from [13] above?

And [15] says "It seemed reasonable to replace both coroutining and pseudo-parallelism by “real” concurrency, using CSP-like input (match) and output (bind) operations on channels (shared variables)."

From theory to practive. I see no way where parallel processes could run in true parallel on a uni-processor machine. Instructions have to interleave on that implementation level. Not even interrupt (processes) are truly parallel. Moving to a multi-core would help. But all software processes need to communicate. If this is done by "send and forget" or "move data during blocking rendezvous", I don't see that any of this would move a process into being not truly concurrent and only interleaving.

See from this programmer's head, when two processes are not communicating on a CSP blocking channel, they are really running truly concurrently! And when they do communicate (and synchronize), that's what they do, and true or pseudo concurrency is not even asked. It's in a different problem domain.

I need help! Even if I, to a certain point, don't care if my runnable processes would be considered pseudo or real. To me I have mostly learnt them to know as both.

However, for formal language tool (like Promela) processes I guess it certainly matters.

(17Jan10)

[C.3] - USL Distributed Event Driven Protocol

This is described at comment [C.1] above.

I got a complete surprise when I saw this: it's quite close to my "The "knock-come" deadlock free pattern" [16]. The wording is different, but the essence is the same: tell that you have something ("knock"), then wait for a reply saying "come" - then send the data.

The USL now, is it a "language around this pattern"?

Since USL seems to build on this pattern, almost everything changes in my comments!

[16] is a back-to-back pattern that I have proven with Promela/Spin to be deadlock free.

And the protocol is why, in USL "There can never be overflow; faster producers and slower consumers are never an issue". Of course! (That being said, a faster producer than consumer is an application level problem which must still be solved at that level.)

Now, is it I that have turned synchronous into asynchronous (my "come" is sent on an asynchronous interrupt channel with no data), or USL that, inside the engine, really is synchronous? They say: we can push the accelerator any time, and I say: but the crankshaft makes ignition synchronous! Are we saying sort of same thing?

Also, when USL says that "[1.2] - To our surprise, changing from a synchronous OS used in unmanned missions to an asynchronous OS in manned missions supported asynchronous development of the flight software as well." - I have to agree! I have said the same of "my" architecture! See [6] "High Cohesion and Low Coupling: the Office Mapping Factor."

So, it's true that the process, communication and synchronization "concurrency building blocks" certainly show up in the final software architecture. A wooden and a brick house look different and have different qualities.

Next, I should study Universal Systems Language built "around" the USL Distributed Event Driven Protocol. I think I have discovered the canvas of this painting. Then I need to find out at which distance I should relate to it. As a reader, viewer or ..painter?

I have now added a comment to [16].

(18Jan10)

[C.4] - Another synchronous/asynchronous comment

The company Altreonic in [17] (above) explains about their OpenComRTOS in a newsletter:

"The asynchronous services are a unique type of service as they operate in two phases. In the first phase the application task (or driver) issues a request to synchronize or pass on data via an intermediate hub entity. This request is non-blocking and hence the task can continue. Later on it can wait for the confirmation that the synchronisation or data exchange has happpened. A task can simultaneously have multiple open requests using one or more hub entities. This is valid for sending as well as receiving tasks and provides a very flexible and efficient mechanism. Safety is assured by the use of a credit system."

My boss at work sent me the newsletter, wondering if this was perhaps in the same street as the "knock-come pattern". I got a feeling of rememberance, because I had talked with people from Altreonic for many years at CPA conferences, from the Eonic days and even before that, when some of us were using transputers, and quite a few occam.

So, this street seems to have several buildings: The USL protocol, The OpenComRTOS and even our knock-come when used as we use it - together with a small, embedded data base. And certainly there must be many more!

I sent a mail to Altreonic and queried. A thread followed with Eric Verhulst, who wrote (he allowed me to quote):

"I remember we had our discussions many years ago when Eonic was promoting Virtuoso. Already at that time there was a discussion whether Virtuoso violated the strict rules of CSP. At that time I called it a “pragmatic” superset of CSP. The work done on OpenComRTOS (whereby we used formal modeling) has proven that this intuition was right. I think it has also "solved" the synchroneous-asynchroneous debate. It is not an XOR debate or an AND debate. At the lowest level of the primitive operations, it is synchroneous. At the higher semantic levels we can introduce asynchronisity. That good thing is this is not in conflict at all with CSP."

What more is there to say? Personally I started this blog with quarreling about sync versus asynch. Like in The Alchemist by Paulo Coelho there was something to return back to: maybe it's not that simple: "sync versus asynch".

What more to ask: "at what level and for what usage?"

There is no use to repeat comment [C.3] above. But maybe the general idea holds as a comment also here.

There is more to the OpenComRTOS story, though: the consequence which their hub has on the safety of the application. The use of the knock-come pattern, use of the USL protocol and the use of the two-phase hub mechanism of OpenComRTOS seem to all help with this. It's "send and not forget" or "send and forget but then be reminded again". For certain needs this is great stuff. Especilally if one needs to send off lots of data or commands to wherever, let them do their anyway(?!), even at much much slower speeds, and then get the result. The idea then is to both report how it went, and not to get empty of memory for this stuff. No crash. Limits are handled by design. OpenComRTOS do static linkage of packets with no malloc, with blocking if there is no packets left.

Alternatively one could have a data bank with enough entries for each request or command and then tag the elements with status, like: want to send down, is expecting reply etc. This is how we did it in an embedded product: with success - but the knock-come pattern was used in connection.

A synchronous channel is really a handle to common data. With space for no packets, the receiver may hold the producer. With zero data there could be an asynchronous "interrupt" channel and asynchronous send. With some space in the channel, it would block when it's full. With Ada rendezvous, data exchange is synchronous (blocking) and bidirectional. The channel concept evolves, it could be one-to-many or many-to-one, and they could be connected to a mechanism like a barrier synchronization mechanism.

However, what I would look for is to have WYSIWYG semantics: no matter how asynchronous the mechanism I use, I would want to only relate to the defined protocol between the concurrent processes. I will not need to know how that other process is coded internally. Think about this: if I hold a channel (or whatever input mechanism) while I do the session I was just told to start (I am a server), am I allowed to send the result back to the originator client that used me, and not think a second about how that would influence the other clients competing for me? And the clients, could they be coded 100% irrelevant of the server code, provided they agree on the interface contract?

Am I allowed to control my clients myself, to make handling "fair"? Do I have the mechanism? Will this system behave the same way if I use a client with a statically linked channel, across the internet, or on a channel sent over a channel. Have a look at occam-pi for these things, look at the Go programming language, read the CPA-someyear literature (Communicating Process Architecture poceesings).

Read papers you seem to disagree with. Study newsletters from interesting companies.

There is much more out there than just send and forget.

(11Feb2010)

014 - Notes about iPhone

2009-07-20T17:23:00.142+02:00

Please observe that this note, even if started on July 20, 2009 has been extended with several sub-posts that are being continuously updated (last in 6Mar11)

Being a software engineer (with some older hw skills mingling about), I am used to collecting lists of errors, problems and ideas. Here is my list for my iPhone. I have an iPhone 3G, quickly updated to latest upgrade from Apple. At the moment I have sw version 4.21, build 8C148, but the problems below have been tagged with actual versions of the time. Please also note that my experience here is with Mac machines, not with say, iTunes on Windows.

I will try to update the list when the problems have been fixed as seen from my iPhone. I also have some other iPhone posts in this blog. My goal is to try to help getting a great mobile phone even better!

Disclaimer: I try to be accurate and not confuse here. But what I say may not be correct for your configuration, iPhone or machine. Some times I do arrive at conclusions, but I try not to jump on them. Still I might have! I would certainly appreciate comments of any serious sort!

iPhone Troubleshooting Assistant etc.

Observe Apple's at iPhone Troubleshooting Assistant. Pages there lead you safely through scary encounters like restoring the phone. Don't be afraid, the procedures are as elegant as the rest. Bear in mind that data belonging to your AppStore are not any "restored", but reappear on your next synchronization. (Norwegian: Problemløsingsassistent for iPhone)

Also notice chapter 7 - Disappearing iPod/iPhone from machine (iBook) here, ref [2].

Newest on top:

16 - iPhone synching picture fails on 3G, not 3GS (open)

15 - Spacebar double-tap to close brackets? (open)

14 - 4.0 GB not sent and 4.0 GB not received (open)

13 - MMS settings changed all by itself! (open)

12 - iPhone screen canals? (open)

11 - iPhone disabling cellular internet access - button missing (open)

10 - Moving iPhone backup files to/from a network disk (closed)

9 - Sending mail via GSM server when home's wifi is out of reach (closed)

8 - Cannot delete SMTP server(s) (closed)

7 - Disappearing iPod/iPhone from machine (iBook) (not really closed)

6 - Setting light intensity (comment)

5 - Could not connect to any operator - SIM card out/in (open)

4 - Cannot connect to network "somename" (closed - was hw)

3 - Mobile data network settings partly lost (open)

2 - Audio memo with "zero size" plays (open)

1 - "Empty" Camera Roll (closed for Mac/Windows users, open? for Apple)

16 - iPhone synching picture fails on 3G, not 3GS

See http://oyvteig.blogspot.com/2010/09/020-mac-os-x-network-disk-partially.html, chapter "Related problem - iPhone synching picture fails on 3G, not 3GS"

15 - Spacebar double-tap to close brackets?

Depending on user settings it's possible to insert a period character (dot) by double-tapping the space bar. This is very nice. After the period it also insert a space proper, making ready for next word.

However, some times I'd like (the same function) instead to close a bracket.

See the use in the above sentence. I have in my mind double-tapped to get the ) and then later on for the ending period.

It's easy to implement this. Just make a priority list and do accordingly. In my suggestion, all closing of any left bracket type has priority to inserting a period.

I'd like to close these type of parenthesis: [ ], ( ), { }, < >.

Easy with virtually no side effect! Most brackets a user doesn't want to close before a period. Of course, it is possible to have a dot inside a parenthesis. Either make a small select box to select the right bracket or the dot, or make an exception for email addresses or url.

This may of course be configurable, as en extra option in addition to the dot insertion.

How about also including other block defining characters, like quotation marks?

Reported to http://www.apple.com/feedback/iphone.html

14 - 4.0 GB not sent and 4.0 GB not received

iOS 4.2.1 8C148 on a 3G iPhone

My iPhone told me I had sent and received 4.0 GB ("Mobilnettverksdata: Sendt 4.0 GB, Mottatt 4.0 GB" in Norwegian):

Having heard about iPhone users with too large bills, I immediately logged into My Account at Network Norway. I had in sum used less than 50 MB (0.05 GB). I have now learnt that if iPhone shows 4.0 GB in both directions, it is probably wrong. But what it it shows other figures?

This was the first time after a couple of years. I cleared the statistics, as I have done every 20th. And it's not run wild yet.

Reported to http://www.apple.com/feedback/iphone.html

13 - MMS settings changed all by itself!

iOS 4.2.1 8C148 on a 3G iPhone

All of a sudden my mobile data settings had been changed! I usually have Network Norway with this set up. All of a it pointed to TIM, a subsidary of Telecom Italia!

It is true, my wife and I were in Italy this summer, and TIM came up as an operator then. But that's 8 months ago! And it does not explain this!

I have changed synching iTunes machine lately (from iBook Tiger to Mac Mini Snow Leopard), but it points to the same network disk. See figure on my Blog 019.

Observe that "all the time" I have had this iPhone, the Mobile Data settings have been volatile, but they have changed to all empty - see chapter #3 below. So, I am used to typing in all the settings, if one could ever get used to that sort. I think it may be after synchs, but I know it's also after restarts, that I've had to retype. (I have a Picture folder called iPhone with screen shots from the correct settings. Nice to have.)

But this time: I have no idea, and have decided not to speculate, just to be open for comments from any one of the below - or from you!

I have reported this to Network Norway (mail), TIM (Olivetti, mail) and Apple (http://www.apple.com/feedback/iphone.html).

Responses

Network Norway: "We know that settings may change when one has been abroad with iPhone. If you have got new settings, and a field is non-blank, it is often because the operator abroad has sent new settings to your phone. However, in the cases where we have seen this, the mobile user has usually been queried for approval." (translated by me) My comments: I have not been queried, and it's 8 months ago and I think the response quite vague. However, I did learn that an operator may update my settings.

Search words in Norwegian

iPhone Instillinger / Nettverk / Mobildatanettverk (Mobildata) MMS forandret av seg sjøl!

12 - iPhone screen canals?

According to Wikipedia [1] "waterway canals are navigable transportation canals used for passage of goods and people, often connected to existing lakes, rivers, or oceans."

An "iPhone canal is a navigable icon transportation canal used for passage and short stop of icons, often connected to existing screens or passing through them". This is my suggestion of a definition of one such.

I'd like to have it!

The figure shows my page 1 at three stages. I have a hand made canal there. It's the open unused space. To illustrate I have filled up my page 2 fully. Usually all my screens have a canal. Let me try to transport the iPod icon (1.2). When it's out of the way, the screen reorders (1.3) - the "Instillinger" ("Settings") fills up the vacant place. I like this, no empty space, shuffle into sequence is the rule. All black space is left at the end.

The problem comes when I want to take iPod past page 2 to page 3. Page 2 is continuously being reshuffled. No, not really: if I keep on pushing to the right, on page 2 it would discover that I want to move on to page 3 before if starts to reshuffle page 2. This is grand design!

However, it's not obvious. And, I cannot take a pause and stop at page 2. It then would spill over to page 3 the last icon. Of course, it would not know that I just took a break.

What I would like is that the next screen would shrink and align to top so that I get a black channel beneath. Should I stop, my icon on its way would just sit in the channel. Should I continue? Yes, I grab it again and push to screen 3. Screen two is normalised on its way out of the visible area.

When I am finished I push the icon up into the shrinked screen, the canal is pushed down and out, and the screen scales to standard wiggling icons and does the spillover if it's necessary.

Another great thing about this is that reshuffling would not happen before I ask it to.

Even when I start to move, that screen would open a canal for me. Fine!

Reported to http://www.apple.com/feedback/iphone.html on Nov. 21st 2009.

[1] - Wiki on "Canal": http://en.wikipedia.org/wiki/Canal

11 - iPhone disabling cellular internet access - button missing (closed)

Thanks, Apple! Fixed on version iOS 4! But I've kept the legacy text:

At the moment, the only way I know of how to disable cellular internet access - to save me from a catastrophic bill when I go abroad with my iPhone - is to set the mobile data network APN name wrong.
My provider tells me to set APN to "internet" (fancy name..). That's OK in Norway, where I can afford the data megabytes. But even moving myself into neighbour Sweden makes the megabyte rate n-fold, in other word: prohibitive. (The reason for this is that the providers milk the abroad market here, and there is not yet any regulatory mechanism to stop them. Also, we're in the pre-competitive market phase, it's simply too early. The iPhone, which has made us want to surf, also when abroad, just hasn't been around long enough yet. And not enough regulators seem to have an iPhone..)

So, abroad I set APN to "internet not in use" - which is a perfectly wrong APN name - and when I get back home again I reset to "internet".

I certainly appreciate the minimalistic iPhone user interface, but this solution does not help me! What about a button somewhere? (3.1.2, 7D11, open)

Reported to http://www.apple.com/feedback/iphone.html on Oct. 17th 2009.

10 - Moving iPhone backup files to/from a network disk - Or: how to restore iPhone from another machine (closed)

When I had to do a synch of my iPhone from the first floor machine connected to the network disk instead of the ground floor machine (which I usually use), I got a problem. I needed to take a complete restore of a previously stored backup file.

Standard synching with iTunes seems to be ok, even if you have to be aware of calendars and address book (and maybe other things which belongs to a machine). Blog 3 gives a flavour of the iTunes network disk world [1].

However, even if the iTunes data base is on the network disk, the iPhone backups are not. They reside locally in your Mac home folder /Library/Application Support/MobileSync/Backup [2].

When I needed to restore from an earlier backup, I moved all the directories there (with names like 5390181f63c8d9bd90bfb99e893792f48cebef08-20090802-114557) from the ground floor machine to the network disk. Good to have for backup of the backup! Then I moved them, or the one I wanted, to the first floor machine.

Perfect, now they were visible in the iTunes' - Preferences - Units on the first floor machine (bare with me if you cannot find things by my names, I have translated the menus from Norwegian). So, now I could "restore without backup" (using the just moved "backup" file!) also on the first floor machine.

It looks to me that "without backup" means "without your backup" or "with iPhone backup of operating system and associated files only".

A warning: Restoring Without Backup is a Serious Thing (it's meant to be - restore and backup are separate things - as is synchronizing). If you want newer Notes notes to survive, copy them first with PhoneView; (or similar, search here) and restore them afterwards. And, data that are local to AppStore applications would not survive - you did choose to restore without backup! So, if you restored with no backup, then no backup of those things will appear. So, the lists you had in Todo will reside somewhere in another backup. But afterwards, you may synch again with your present music, movies, pictures etc.

Still, it's not that servious to restore without backup! You could restore with backup of an earlier version and do screen cuts and save to the notes, or (if it's 3.0 or newer), cut and paste data from the applications and send offas mail or paste in a note (ref. PhoneView). Just find a way to store if (be it pen & paper if you must!) Then you could restore without backup and paste things in again more or less by hand. So, the name you had put in there isn't as losta s you thought. It was recoverable, as long as you have a backup - which iTunes does for you when it wants!

Still, I think that restoring without backup is advocated too often. At least, it's my experience. Do try to avoid it!

I am afraid that parts of this chapter may be vague or confusing. Whenever I know better, I might try to do a rewrite.

[1] - 003 - Shared iTunes library on a network disk
[2] - Scanning your iPhone Backup Files

9 - Sending mail via GSM server when home's wifi is out of reach - (closed)

After receiving my "swap" iPhone (point 4 here), I goofed the SMTP outgoing list, even if I had saved the setup (point 8 here) screens!

Earlier, the iPhone switched to GSM mail when I left home's safe wifi network. Now I got a response saying that the mail server did not like the address I sent to. But it was my own.

When I called my GSM mobile provider, they said that I did not need the user name and password. They were almost right, because when I switched off the primary SMTP server (most often used from home), sending mail did work. But enabling the primary server again, it failed.

To make a long story short: when I included user name and password for both primary and secondary SMTP servers (even if they individually did not seem to require this), then the failure of the primary's attempt did not stop the process - but went on to try the secondary as well - for success. (This is how I "read" the behaviour.) (3.0.1, 7A400, closed)

8 - Cannot delete SMTP server(s) (closed)

The list of my outgoing mail SMTP [1] servers had added up - and for some days I could not figure out how to delete the unnecessary entries. I only needed my wireless SMTP server as the primary, and my GSM or mobile provider as the only secondary.

However, while fiddling around (during point 9, above), the primary server appeared three times, just between primary and secondary! And when I opened them, they were grayed out. Trying to disable them, they looked enabled again - even if they weren't.

So, how did I clean the list? Here is a recipe that worked for me (inspired by [2]):

First, take a back-up of your settings. The easiest right there may be to take screen snapshots of all the correct SMTP settings. And if you need to note on the pictures, install Sketches from AppStore. (Then, when you synchronize pictures, upload these to the laptop camera upload directory and make a new iPhone pictures directory called iPhone, and move the configuration screen shots to that directory. You'll regret it so little that you will expand to several other set-ups. (You take screen shots by pressing the middle and the off button simultaneously.))

Then, back to the ugly SMTP list. Open the primary, or any one that you are allowed to edit, i.e. not grayed out. Presumably you should try with the entries you want to remove. If the red "Delete server" button is not there, empty the host name field, go up one menu level and open it again. Then the iPhone realises that an SMT server with no name is deletable, and the delete button should appear. Use it. (I had noticed before that the iPhone uses this behaviour: you'll have to make a field empty, then a branch in the menu tree happens - with new buttons arriving.)

Repeat this procedure until you're done. Then add the SMTP server(s) that you had to delete but really wanted to keep - since there is no way to slide table entries up and down. As I said, this certainly worked for me. (3.0.1, 7A400, closed)

[1] - SMTP server: http://en.wikipedia.org/wiki/SMTP_server

[2] - http://discussions.apple.com/thread.jspa?threadID=1711342 - see esullender 30. Jan. 09

7 - iPod / iPhone disappeared from machine (iBook) (not really closed)

When I needed to synch my new "swap iPhone" (note 4 here), my Tiger PPC iBook had decided not to see it! It was not seen by any application, be it iTunes, iPhoto, Image Capture or PhoneView. The iPod also was lost!

However, I noticed in System Profiler [1] that the iPhone was seen by the lower layer USB driver. I observed that it was misplaced on some occasions, and I even could find a configuration where it was not seen. I don't exactly remember how, but I did a mix of the two iBook USB connectors and a hub, with the hub, an external mouse and the iPhone in combinations. Cabling around and updating the display gave me some scenarios. I imagined: there must be some invalid USB descriptors(?) in there! But I had tried several times to restart the machine.

I was pressed. Even if the Intel Mac Mini worked, I did nat want to use it for iPhone work. And the PPC Lamp Mac worked, but with USB 1.0 it took too long. I needed the iBook up! Observe that iTunes with a network disk has been described in post 3.

I thought that maybe the operating system would clean the USB table if I restarted the machine with the iPhone connected! I did, and the soothing beep from the iPhone came twice during power up! And the rest... is just bliss! All is fine now!

I don't know why this happened. I had done Security Updates 2009-003 and 2009-004 on the iBook Tiger PPC in the meantime. But I have no reason to believe that this could be the cause. Or maybe I had plugged the iPhone into the unpowered USB hub [2]? There certainly must have been some reason?!

Update 22June2010: This problem is still persistenet with iPhone 3G with 3.1.3 and iTunes 9.2 (61) running on iBook with Tiger Mac OS X 10.4.11 (8S165). Same solution as above, though!

Update 1April2010: iPhone picture upload from Camera Roll failed to start Image Capture. All of a sudden this happened, the iPhones were visible in iTunes and iPhoto, but even after all combinations of plug, unplug, restart etc,. the Camera Roll (3.0.3 (333) Mac OS X Tiger 1.4.11) did not start. My fix was to start PhoneView (2.3.2) and copy the pictures over manually, and then delete the whole DCIM camera folder, including sub-folders .MISC and 110APPLE. On my iPhone, several other sub-folders 111APPLE up to 120APPLE had appeared. Some contained pictures, some not, but all pictures were visible in Camera Roll. On my wife's there were only the two sub-directories. For both iPhones, PhoneView helped. After I had deleted the directories, I stopped PhoneView and unplugged the iPhone. Then I filled a picture into Camera Roll and plugged the iPhone in again to find that Image Capture started again! (Filed 1April10, but it's no hoax!) (iPhone 3G and 3GS 3.1.3 (7E18))

Filed to http://www.apple.com/feedback/iphone.html

[1] - Mac OS X System Profiler: en.wikipedia.org

[2] - In Troubleshooting and Maintenance - iPhone: The Missing Manual by David Pogue: http://oreilly.com/iphone/excerpts/iphone-themissingmanual/iphone-troubleshoot-maintenance.html - chapter "Doesn't Show Up in iTunes" it reads "don't plug it into an unpowered USB hub."

6 - Setting light intensity (comment)

I wish setting the iPhone light intensity would be faster. I wouldn't mind if the last screen, with the slider and the Auto switch was gone. Here is a naïve suggestion, where it's controlled in the Settings screen directly:

Fig. - (Norwegian: Instillinger/Settings, Lysstyrke/Light Intensity)

Of course Apple has thought about this. They don't want the graphical and functional overload of this suggestion, or anything similar. But maybe they would be able to handle the text in different languages (lengths) better than above? And the extra Auto button of the last screen - here, with no text, or some pop-up - combined in the slider, where my suggestion is to push it down and then slide it in lower position.

In some AppStore apps one just slide the finger up and down on the screen to get it lighter and darker. But if the SDK API allows a user to do this, then Apple can't (or will not) take it over as a system wide command? (Comment to 3.0)

Filed to http://www.apple.com/feedback/iphone.html

5 - Could not connect to any operator - had to remove and insert SIM card (open)

After I had an incoming call perfectly fine, coming home I sent an SMS. But it failed! The exclamation mark was shown. A resend did not help. I tried to call, but failed. Initially with no warning, but down in this procedure - with a warning. I could not be called. I had no Phone, just the i.
I looked at the service providers, they were three. The list looked correct. One of them was my wife's provider, which worked for her. We tested it. The Auto mode did not connect to any, and the forced mode seemed like it would connect, but it did not.
I switched the telephone completely off and on: no help. Then I switched off, took out the SIM card, inserted it again and switched the iPhone on. Worked!
Now I'll carry the supplied SIM card removal pin in my wallet.

I don't know to which degree any mobile could exhibit this behaviour from time to time. The program base from which the code is taken, and the degree of control when gluing that sw and some new together - for that particular phone, would probably never yield a 100.000% stable product. Now I am uncertain as to which side of the decimal point my phone is. However, hoping for updates does help.

Disclaimer: I do not know if, by coincidence, my service provider was down for that very period. That probability is very low. (3.0, 7A341 no fix yet - point 4 (below)-not done yet)

4 - Cannot connect to network "somename" (closed - was hw)

My iPhone just got unwilling to connect to any wifi (wi-fi) network - even if it displayed the list of possible networks perfectly fine. This most probably is a hardware failure, so it will be returned for repair. The Apple reseller agreed. My guess, as an old hw engineer, is that it's the wifi sender that waved goodbye.

I searched in the Apple support pages and on the net to try to get some tips. So, I reset the network settings in the iPhone, restored the iPhone from iTunes (both with and without backup (bear in mind that data belonging to your AppStore are not any "restored", but reappear on your next synchronization - provided you have a backup.)) - but nothing helped. Switched airplane mode on and of. Removed wifi password and tried different wifi locations that I knew had worked a few days earlier - it still could not connect. Beware, I always keep my iPhone in protective silicone and am careful with it.

I have a feeling that bluetooth also has died, but since I normally don't use it, I am not certain if the connecting problems there are finger trouble. (My iPhone serial number 85838VNMY7H, model MB046LL)

Result: I got a new "swap iPhone" back from service. (It's serial number 5K9276D6Y7H, model MB489B). Observe it's a new model number as well. It now works!
I delivered it to Eplehuset here in Trondheim on a Monday (in August 2009) at 12, it was received by Moobi in Oslo on Thursday at 17 (3 nights!), and the new was sent on Wednesday the week after and arrived here on Thursday (1 night!). That's about 11 days (*). I should have, but was not able to track what happended on Moobi's service page (it was not updated even if I received an SMS telling me it should be). I called Moobi on the phone and received both very supportive responses and one who said he definitively was not allowed to answer me (it should go through Eplehuset, he said, which would have been void, because when I asked them they told me to call Moobi). I called back and hit a different person.
(*) 11 days, when Eplehuset could have swapped the phone over the service department's counter while I waited, or delivered it 1 day after! However, my wife's old 1999 Nokia surviving in a box saved me those days, since my 2004 Sony Ericsson (in the same box) did not like my iPhone SIM-card.
(Norwegian phrases: feil på iPhone, repare, reparert, fikset, wifi sender funket ikke)

3 - Mobile data network settings partly lost (open)

Some times the mobile data network settings disappear, so I have to reenter. This also goes for the MMS settings. I have learnt it now, it's only three easy lines of text, but it's irritating. However, in case I forget, all the settings have been screen clipped and synched and moved into an iPhone picture album. So it's always with me! (3.0, 7A341 no fix yet)

2 - Audio memo with "zero size" plays (open)

I did some recordings the other day. In the list of memos, some display as length 0 (zero, null) seconds. But they play correctly, and the time progress line shows correct length. In PhoneView (point 1, below), size comes out as "unknown". (3.0, 7A341, no fix yet)

See http://discussions.apple.com/thread.jspa?threadID=2086056&tstart=0

1 - "Empty" Camera Roll (closed Mac/Windows users, open for Apple!)

Right now the Camera Roll tabular view (with pictures in a table) looks empty! However, the count of pictures and the single thumb, in the list of all albums is right. And all pictures upload correctly - so that I may see them for the first time. So, I have a film type camera at the moment (not 100% true, though: shooting pictures very fast may help..). This all happened after - you tell me? See http://discussions.apple.com/thread.jspa?messageID=9850890#9850890
(3.0, 7A341, closed for me, open for Apple!)

Apple could have saved us iPhone users of this unnecessary frustration. The iPhone operating system could have discovered that the user "has put a picture into Camera Roll, but it displays none ..oops ..rebuild contents database (xml file?)". But then, in the first place, why does the operating system allow a program to corrupt a (database) file?

Reported to http://www.apple.com/feedback/iphone.html

Fix (Mac)

I discovered several places where this has been discussed over the years, a notable one being [1]. But I stopped at http://www.zeldman.com/2007/11/11/iphone-camera-bug/, Jeffrey Zeldman's blog. There, "Ben said on 24 June 2009 at 4:48 pm" (search for that text) pointed to the solution. There, Ben points us to PhoneView at http://www.ecamm.com/mac/phoneview/ by Ecamm Network. I downloaded the free trial version and did Settings, Show Entire Disk. I did like Ben had suggested and deleted the /Disc/DCIM/ folder on my connected iPhone. This is the same exercise that Finder allows me doing on any other camera (and I seem to remember that the iPhone's camera has been visible in Finder earlier?)

Now, Camera Roll again works as expected, it's empty and ready to be filled again!

Ooops... I experienced a problem with PhoneView, where it did not read Notes files etc.. I don't know if it in that situation would have failed to delete /DCIM. However, Ecamm sent me a procedure to rectify: quit PhoneView and clear its preference file at Home/Library/Preferences/com.ecamm.PhoneView.plist. Then do an iPhone/iTunes sync, and restart PhoneView. I have not verified this procedure, but post me if this helped get Camera Roll on the air again 2nd attempt.

[1] - http://discussions.apple.com/thread.jspa?threadID=1183455&tstart=0

Fix (Windows)

Ecamm Network (above) recommends TouchCopy for Windows users (see http://www.wideanglesoftware.com/touchcopy/). NeoDoc has confirmed that TouchCopy has directory delete functionality, and that - with the iPhone directory /DCIM gone - it works. But do remember to load the pictures to the PC first! Thanks, NeoDoc!

(Norwegian search tags: "problemer med iPhone, feil, virker ikke")

If you want to comment, please refer to section numbers!
.
.

013 - Overview

2009-07-12T16:35:00.052+02:00

Posts on http://oyvteig.blogspot.com, (http://oyvteig-2.blogspot.com) and [http://oyvteig-3.blogspot.com]
Goto Archive and about

 013 - Overview

[037 - Märklin Swiss electrial locomotive SBB Ae 3/6 II H0 models]
 036 - WYSIWYG semantics
 035 - Channels and rendezvous vs. safety-critical systems 
 034 - Output guard vs. "channel ready" channel 
(033 - A TV rack that isn't)
(032 - Pull-up/lift-up curtain)
(031 - Wooden window trickle air vent cover)
 030 - Burning an mp3 music CD from iTunes
(029 - From ceramic vase to cutlery drainer)
(028 - DIY prototype of an iPad 2 holder for car glove compartment
 027 - Experiencing Apple AirPlay
 026 - iOS picture sorting and file system do not tango
 025 - iPad2 on vacation, ups and downs
 024 - 16:9 video aspect ratio with width 720 uses height 404 instead of 405
 023 - Concurrency: one writer and one reader is bad enough
 022 - C1X and C++0x concurrency in C and C++ Working Drafts
 021 - The problems with threads
 020 - Mac OS X network disk partially seen
 019 - Play (with) network-disk based iTunes library?
 018 - From concrete CDs to abstract iTunes files. And now the old amp broke
 017 - Safari and Opera Mini on iPhone/iPad
 016 - Cooperative scheduling in ANSI-C and process body software quality metrics
 015 - In search of Go! programming language rendezvous
 014 - Notes about iPhone
 012 - Wishes for a folding editor (2)
 011 - When is my mail really outgoing?
 010 - A running Promela model
 009 - The "knock-come" deadlock free pattern
 008 - Safari oblivion - Firefox to the rescue
 007 - Synchronous and asynchronous
 006 - Preferential programming versus me (in work)
 005 - Press '?' for help, press '!' for new idea
 004 - Living with a network disk in the house
 003 - Shared iTunes library on a network disk
 002 - Hjemmeruter og CanalDigital (in Norwegian)
 001 - iPhone Notes asymmetry (case closed)

.
Disclaimer: Most of these posts are related to private technology use and computer science, along my line of interest and experience. Some notes refer to papers published in my name at the named company I work for. These are approved and public papers.
.
.

012 - Wishes for a folding editor (2)

2009-07-03T17:19:00.002+02:00

This post is meant for us to collect thoughts. I would appreciate anyone to comment here, and hope that we could collectively bring about a spec for a new folding editor. Maybe a programmer would emerge, too?

Starting point may be seen as my Designer's note [1].

Observe that my favourite folding editor is WinF (or F or WinF32). It's freely available at [2], as Windows exe and source files. At the moment it just is best..!?

Read about folding in [3].

[1] - "Wishes for a folding editor" (1): http://www.teigfam.net/oyvind/pub/notes/11_Wishes.html
[2] - "Download winf folding editor by Julian Wilson": http://www.teigfam.net/oyvind/extern/winf32_folding_editor/index.html
[3] - "Folding editor" at Wikipedia: http://en.wikipedia.org/wiki/Folding_editor

011 - When is my mail really outgoing?

2009-04-07T15:15:00.007+02:00

I would like to know about outgoing mail

I have this interesting case that when I am a visitor I am not allowed to send outgoing mail with my mail client (Mac OS X Mail 2.1.3), but I am with my iPhone (any version, including 3.1.2 October 2009). Both, over the local wireless network at the places we visit.

Disclaimer: sending and receiving mail with a net browser it not the theme of this post. That always works, from any location. I guess - since I am a known user.

Also, if I try sending mail from an iPod under these circumstances, I am neither allowed. With same mail set-up as the iPhone (iPod any version, including 3.1.2 October 2009).

At home

There, the outgoing mail server is smtp.online.no. All users of the wireless which uses that server with my name and password could set up their clients to send in my name. All machines on the net, including iPhone.

Visiting

The network at a friend we often see has smtp.bostream.nu as outgoing mail server. Since I want to send as myself, I have not asked for the friend's log-in details. So, Mail "refuses" me do send over that server, and trying to use the same set-up as at home it refuses me to send over smtp.online.no. I guess it's because the latter server knows the ip address or network provider I use, and then it knows who is sending: it's me, at home, I have paid for the connection: all is well. So, it's not Mail that "refuses", it merely reflects the outgoing mail servers' responses.

My queries

But why am I allowed to send via the visiting wireless from my iPhone (and neither iPod nor Mail)?

Observe that I have told the iPhone not to use the telephone connection by giving it a rubbish APN name - and configured the iPhone so that the telephone provider's server is second to the wifi server. I can see on the mail I this way am able to send from the iPhone that it's routed from bostream's dsl.bredbandsbolaget.se directly to my mail address's incoming server at start.no. My iPhone telephone provider does not seem to be in the expanded mail heading anywhere.

What are the (protocol) mechanisms?

Apple support discussion

I have posted a copy of this post at http://discussions.apple.com/thread.jspa?threadID=1969325&stqc=true

010 - A running Promela model

2009-03-18T21:58:00.004+01:00

Why does this code run and don't deadlock?

mtype = {M_UP, M_DW};
#define LEN 0

chan Chan_data_down = [LEN] of {mtype};
chan Chan_data_up   = [LEN] of {mtype};

proctype P1 (chan Chan_data_in, Chan_data_out)
{
    do
    ::  Chan_data_in  ? M_UP -> skip;
    ::  Chan_data_out ! M_DW -> skip;
    od;
};
proctype P2 (chan Chan_data_in, Chan_data_out)
{
    do
    ::  Chan_data_in  ? M_DW -> skip;
    ::  Chan_data_out ! M_UP -> skip;
    od;
};

init
{
    atomic
    {
        run P1 (Chan_data_down, Chan_data_up);
        run P2 (Chan_data_up,   Chan_data_down);
    }
}

009 - The "knock-come" deadlock free pattern

2009-03-15T16:03:00.048+01:00

Intro and disclaimer

This note is based on a published paper [1]. At the time of publication (2005) I did not think that chapter 10 would describe any more than an interesting case of software engineering. To this date it stays so. However, people have asked me about this pattern. This note therefore describes nothing new, but will add some more discussion.

Disclaimers: 1.) Should anybody claim this pattern as already used, invented or even patented, prior to 2005 - good! It is not known to me (but now I do know that I am not alone, see bullet list below). Please mail me if this is the case - so that I may be abe to honour it here - or obsolete - at least the suggested name. 2.) Even if the pattern is deadlock free, there may be errors in this note. 3.) Any usage of this pattern is out of my control and responsibility.

In Jan.10 I discovered that the "USL Distributed Event Driven Protocol" seems to be similar. See [4].
In Feb.10 I was made aware that the OpenComRTOS also uses a mechanism in the same street. See [5].

I have named the pattern "knock-come" for a rather obvious reason - the explicit behaviour it must display in order to function properly. The name does not fully describe the behaviour, but it should hold as a reminder.

Rationale

When two software threads or rather, processes, need to talk with each other at the same time, we would get a deadlock if they were communicating over synchronous, blocking channels. This note describes a simple software pattern that removes the possibility of deadlock. It also shows a formal model and verification of the algorithm, written in Promela and evaluated with Spin [2].

The original description, from [1] - of the "knock-come" pattern:

10. Deadlock avoidance

Where two processes spontaneously need to send data to each other, a blocking communication scheme might cause deadlock. This is a state where processes try to communicate – in a circular pattern – blocking for each other to become ready, and therefore unable to proceed. This is pathological and must not happen. All processes run at the same priority, so any priority inversion problem is ruled out in the system.

Figure at http://www.teigfam.net/oyvind/pub/ercim05_at_euromicro-2005/Figure_chapter_10.jpg from [1].

The pattern we chose to avoid this was to give the processes clear roles: slave and master. Master may send on the blocking channel any time (solid arrows) when it has something. Slave would never block on any spontaneous message, since the asynchronous “poll me” message (stippled arrow) lets the slave go on and not block, and then instead hang in an INPUT or ALT on the input channel. When the “OK, come on” message arrives from the master, the agreed upon protocol assures no deadly embrace.

Software architecture

Figure at http://www.teigfam.net/oyvind/pub/CPA2006/presentation/Fig1.jpg, from [3].

We see two processes, the top and bottom P_.. processes. The left dotted lines, downwards are asynchronous non-blocking send channels. One one of them is of interest now. Let's call it the knock channel (Chan_202 in the figure). It carries no data.

The other arrows are one up-going come channel (Chan_201) and the down-going channel (Chan_200). Both are synchronous non buffered and blocking send and receive. Both carry data, together with a tag that describes the contents. They are drawn like a two-way channel, but they are two one-way channels. In addition to the come message, these channels are also used to send the actual data - in both directions.

Message sequence in words

Whenever the bottom process needs to send data, it may do it any time. It is the master.

Whenever the top process, the slave, needs to send data, it may send one only query ("knock") any time. For the 1.) data up and the 2.) query - not to deadlock, the query is sent as a no data "interrupt" signal. The slave would send only once per session, so P1_Slave will always proceed (and not block) on this sending.

Whenever the bottom process decides that it has time to handle the query, it sooner or later sends the "ok, come on" up and immediately waits for the slave to respond to that message by sending the data down. Whenever the data has left slave, the sequence with the query may be repeated.

By sooner or later we mean that in the meantime the master may send any number of messages up. As we started with saying, the master may send any time.

The reason why the knock /query / interrupt channel has no data, is really that is simply that it does not need to have any data. Also, in real life, it is simple to implement an asynch channel with zero buffer.

Formal proof of deadlock freedom

The Promela model code below verifies to no "invalid end state", using spin / xspin. This means that it will not deadlock. Observe that with LEN set to zero it will deadlock. It has been modeled to resemble how the processes are coded with a standard synchronous methodology (language or library) - where only inputs may be ALT'ed over. (Note 010 above gives a very interesting deadlock free alternative.)

The code is my initial iteration on the theme:


mtype =
{
    M_KNOCK_DIRDOWN,
    M_COME_DIRUP,
    M_DATA_DIRUP,
    M_DATA_DIRDOWN
};

#define LEN 1 /* Invalid end state (ie deadlock) if this is zero */

chan Chan_knock_down = [LEN] of {mtype}; /* Chan_202 */
chan Chan_data_down  = [0]   of {mtype}; /* Chan_200 */
chan Chan_data_up    = [0]   of {mtype}; /* Chan_201 */

proctype P1_Slave (chan Chan_knock_out, Chan_data_in, Chan_data_out)
{
    bool Knock_send;
    bool Knock_sent = false; /* necessary? */

    Run:
        if
        ::  Knock_send = true;  /* non-deterministic setting to true.. */
        ::  Knock_send = false; /* ..or false */
        fi;

        if
        ::  (Knock_sent == false) && (Knock_send == true) ->

            /* Am slave, but want to get rid of my data now */
            Chan_knock_out ! M_KNOCK_DIRDOWN;
            Knock_sent = true;

        ::  else -> skip;
        fi;

        if
        ::  Chan_data_in ? M_DATA_DIRUP -> skip;

            /* Am slave, always able to receive from master */

        ::  Chan_data_in ? M_COME_DIRUP ->

            /* Got "come" from master, do it */
            Chan_data_out ! M_DATA_DIRDOWN;
            Knock_sent = false; 
        fi;
        goto Run;
};

proctype P2_Master (chan Chan_knock_in, Chan_data_in, Chan_data_out)
{
    bool Knock_received = false;
    bool Data_send;

    Run:
        if
        ::  Data_send = true;  /* non-deterministic setting to true.. */
        ::  Data_send = false; /* ..or false */
        fi;

        if
        ::  (Data_send == true) ->

            /* Is master, may always send */
            Chan_data_out ! M_DATA_DIRUP;

        ::  (Knock_received == true) ->

            /* Send "come" and get data, according to "contract" */
            Knock_received = false;
            Chan_data_out  ! M_COME_DIRUP;
            Chan_data_in   ? M_DATA_DIRDOWN;

        ::  else -> skip;
        fi;

        if
        ::  Chan_knock_in ? M_KNOCK_DIRDOWN ->
            /* Slave "knocks" */
            Knock_received = true;

        ::  timeout -> skip;
            /* global "else" to make "Chan_knock_in" input a poll */
        fi;
        goto Run;
};

init
{
    atomic
    {
        run P1_Slave  (Chan_knock_down, Chan_data_up,   Chan_data_down);
        run P2_Master (Chan_knock_down, Chan_data_down, Chan_data_up);
    }
}

Daisy-chaining this pattern

This pattern scales. You may take a slave's Chan_data_down and connect to another's Chan_data_up, and let a new master have one more Chan_knock_down from each new slave. This daisy-chains with loop-back, equal slaves, which must then share via-traffic for the others. I have proven it deadlock free with a slightly modifed Promela model.

References

[1] - From message queue to ready queue. Øyvind Teig, 2005. Subtitle: Case study of a small, dependable synchronous blocking channels API. "Ship & forget rather than send & forget". Ref chapter "10 - Deadlock avoidance". See http://www.teigfam.net/oyvind/pub/pub_details.html#Ercim05 and read at http://www.teigfam.net/oyvind/pub/ercim05_at_euromicro-2005/paper.pdf. In First ERCIM Workshop on Software-Intensive Dependable Embedded systems. Editors: Amund Skavhaug, Erwin Schoitsch. ISBN 2-912335-15-9, IEEE Computer press

[2] - Promela and Spin. See http://spinroot.com/ and http://en.wikipedia.org/wiki/Promela

[3] - No Blocking on Yesterday’s Embedded CSP Implementation. Øyvind Teig. Subtitle: The Rubber Band of Getting it Right and Simple. In Communicating Process Architectures 2006, Peter Welch, Jon Kerridge, and Fred Barnes (Eds.) IOS Press, 2006, pages 331-338, ISBN 1-58603-671-8. Read at http://www.teigfam.net/oyvind/pub/pub_details.html#NoBlocking

[4] - "USL Distributed Event Driven Protocol", described in Universal Systems Language: Lessons Learned from Apollo. Margaret H. Hamilton and William R. Hackler, Hamilton Technologies, Inc. In IEEE Computer, Dec. 2008 pp. 34-43
("USL"). Also see my 007 - Synchronous and asynchronous, where this is discussed thorouhgly.

[5] - Comment [C.4] of 007 - Synchronous and asynchronous

--

008 - Safari oblivion - Firefox to the rescue

2009-03-01T15:36:00.015+01:00

I did something legal and lost both new and old Safari

Today (March 1st 2009) I downloaded Safari 4.0 beta for Tiger, to run on the iMac G4 (800 MHz) and iBook G4 (1.33 GHz) - both running Tiger 10.4.11 and security update 2009-001.

The conclusion now is that the on both machines, I have no working Safari browser! The beta crashed on both machines, and the old Safari did not work on either - I started Firefox and downloaded Safari 3.2.1 for Tiger to be reinstalled.

The good tip is to have more than one browser installed on your machine! Should the browser you updated fail, you'd have an easy road to the net. (There is ftp on any machine, to download new versions, or you would have more than one machine or friends with machines, and a memory stick..)

Safari.crash.log

See next - Safari discussion group:

This note at Safari discussion group

I have added this note to the Safari discussion group at Apple's site. Some quite good comments there: http://discussions.apple.com/thread.jspa?messageID=9084774&#9084774

Solution

1.) Delete the Safari.app (it's probably a mix of 3.2.1 and 4.0 beta?)

2.) restart the machine

3.) install with Safari3.2.1Ti.pkg (downloaded with Firefox)

4.) run it for a day, install another browser, then go ahead with Safari 4.0 beta

Cause of problem - and a suggestion for Apple

Prior to installing the 4.0 beta I had moved the original Safari.app to a subdirectory and even renamed it. The Installer got confused - without warning me and do appropriate action - and the unInstaller also got confused (see Apple discussion thread).

Apple should enhance their install software (script) not to err (so fatally for some who would not have an alternative browser) - since it's completely legal to move and rename applications.

006 - Preferential programming versus me

2009-02-22T18:29:00.017+01:00

This post is in work

My goal with this post is to try to position paper [1] and [2] and my own programmer's life, perhaps with the table below. The table may be removed later, right now I will see if it helps me even writing this post.

	(un)real disagreement
I prefer	Why?	!	Why not?
	Why not?	!	Why?	They don't prefer
Doesn't interest me	Why?	agreement	Why not?
	Why not	agreement	Why?	Doesn't interest them
I don't prefer	Why?	¡	Why not?
	Why not?	¡	Why?	They prefer
	(un)real agreement

References

[1] - Epistemological Pluralism and the Revaluation of the Concrete

Sherry Turkle and Seymour Papert

http://www.papert.org/articles/EpistemologicalPluralism.html

[2] - On the difference between analysis and design, and why it is relevant for the interpretation of models in Model Driven Engineering

Gonzalo Génova, María C. Valiente and Mónica Marrero
http://www.jot.fm/issues/issue_2009_01/column7/

[3] - Epistemology

http://en.wikipedia.org/wiki/Epistemology

...

005 - Press '?' for help, press '!' for new idea

2009-02-09T19:35:00.013+01:00

This post is a copy of a the initial (27Aug08) version of a note that was first published at the "Designer's Notes" at my home page (press picture on top of this page..).

..or right at http://www.teigfam.net/oyvind/pub/notes/17_Innovating_in_a_window.html

----- copy begin -----

Innovating in a window ("winnovating")

Always present '?' is for help, but why isn't '!' also always there?

Online help systems [1] have been at our fingertips for many years. I remember, back in the early 1990's that we started to use a "help editor" to create a "help" for the Windows systems that we shipped to customers. The systems were clients, doing man-machine communication, with buttons and menues, and nice curves of pressure samples from rotating diesel engines. The embedded data collecting and computing units were mounted in the ships' engine rooms, while the PCs were in the engine control room. And, the intricacies of the system were detailed for the user, available by pressing the ? question mark button. The help was context sensitive, so that one should get the info needed right then. Not more, not less.

Over the years, we have become accustomed to a computer based unit's context sensitive help system, whenever there's knowledge to reach for.

But, how about the opposite? Some new idea or additional information, some small or large innovation we want to jot down, now! The other direction from the help system, so to say, seen by an ! exclamation mark.

Implementing such an exclamation mark would be difficult. It would contain tools, complex logistics and legal matters. At least the suggestion outlined here.

We talk about context sensitive innovation "in a window". Cast in an idea-editor, especially made for the purpose. More that the "send us feedback" menu entry often found.

When we have jotted down the idea, we would need to store it. An idea-server would handle this for us. It would store the idea where it's supposed to. Like, on my local machine.

Or, you may want to send this idea to the maker of the product or anyone who could handle it.

Sending off good ideas right now is a black hole. The recipient could wrap the idea to oblivion. Therefore, we need an idea-agency or idea-bank, a public and safe place where the idea may be posted. The service should be free, since an agency would get neutral money. The people has understood that it's for the best of all, to have an idea-boiling society.

Now, when a user is able to take advantage of the idea, its source is cast in iron. It's me, and it's my idea that's used. For the user of the idea, "not invented here" could become an asset. "Idea! inside" labeled on the product could display this. People would love these products.

And the user would easily find ideas, through some kind of global idea-portal.

The system should be able to inspire any small or big idea. Jot them down, send them, cast them in iron. And idea-sense would collect money for used (and browsed) ideas.

Of course, this would be sitting above the world's patent system (or..below?), like PayPal uses the banking system to make it easier for the payer.

Did I say that the idea-bank should have an idea-server to handle new ideas coming from reading another person's idea? Stored locally at my machine only, or sent externally to the idea-bank.. So, idea-branching should be catered for. Ideas seldom appear in vacuum. Is an idea the most recursive cognitive pattern in the universe?

Humankind certainly needs to come up with some new ideas. We can't afford to waste one of them!

Except some, that would not serve us..

---

This naive set of ideas I have never seen before. I have made a pdf of the original version of this file at 17_Innovation_in_a_window_27Aug08.pdf virtually becoming my safe? idea-bank. Should this not be new, it would certainly be against my knowledge. If so, please mail me. Should my use of words in this note infringe on any trademark etc., rememeber that I use them in a descriptive context only, not pertaining to any such registered use.

This idea is open for use by any. But (disclaimed above): don't think it's your idea! So, give me what's my part of it!

However, it would need much de-naivification!

[1] - http://en.wikipedia.org/wiki/Online_help

----- copy end -----

004 - Living with a network disk in the house

2009-02-01T21:47:00.080+01:00

Intro

Updated 6Nov2011.

Chapters 1-8 (before Nov 2011)

Our network disk was connected to an Apple AirPort Extreme, version 7.3.2 (1Sept09: 7.4.1 (after 7.4.2 failure (*)) - but the problems below still exist), over its USB line, with no hub. Mac OS X Tiger 10.4.11 on this machine. In this post I will discuss aspects around having a network disk.

(*) If I ever get any of these problems after the update from 7.4.1 to 7.5.2 on Dec. 2010 (and Oct. 2011!) (where Apple seems to have done a version quantun leap) I will update here. If not, consider points closed. New: follow on 004.6 - Airport Extreme version juggling

Chapters 9-.. (after Nov 2011)

Apple Airport Extreme with external network disk over USB replaced by Apple Time Capsule with internal NAS disk.

1 - Disk write access lost

Today I ran iTunes (as we're getting used to now, against the network disk as described in the previous post) when iTunes said something like this: "Could not store the iTunes Library file. Unknown problem (-50)."

I thought that this was a multi-access problem, even if no of the other machines were on. Maybe they had left the file in as read-only? I checked the named file on the network disk: nope. I saw no problem with other files, either.

But when I tried to make a new directory on the network disk, I also got the (-50) error! Some access rights problem for the whole disk I soon learnt!

I started the Airport tool (5.3.2) and disconnected "all users", and disconnected the USB cable for some seconds and then connected again. Then I mounted the external disk again from this machine.

Problem gone!

Here is my recipe to reproduce this error (there may be others): My Tiger iBook is connected to the home network via cable. Airport is switched off. The network disk has, prior to this, been automatically mounted with the Apple Airport disk tool (1.5). I then unplug the cable, and Finder (or the disk tool using Finder?) spins around to eventually discover and tell me that the network disk connection is lost. During that waiting period I try to use the network disk, which is still to some extent visible in Finder - or it may have been that I plugged in the camera, which Image Capture did not see during that time. When I later plug in the cable and mount the network disk manually, write access to it is lost.

2 - iPhone crashed when iTunes got (-50)

Incidentally, my iPhone (2.2.1) was connected to iTunes when I got the (-50) error. To my surprise it crashed and restarted when I had an unacknowledged error (-50) window waiting for me. And it crashed again! And again!

I am not 100% positive that I have seen cause & effect correctly here, but until I see it again this would suffice.

Why did it start again without asking for the telephone pin code? I though this was strange, so I switched the iPhone nicely off and on again. Then it asked for the pin code.

There is a history to the iPhone here. It had, prior to this, restarted because I had received 8 MB of attachments in a mail, that I tried to delete while it was spinning to input.

When problem 004.A was solved I synchronized the iPhone. It now seems to run fine!

3 - Backup of the network disk

Observe that if you have a network disk, you are bad off if it dies on you. You only have that copy!

Observe that the longer a disk has been working fluently, the shorter it is till it fails!

There are several things you may do to soothe the situation, should this happen. Here I list what I do, there are a hundred different other ways to do it.

Pictures, f.ex. I synchronize between the network machines, so that at least one of them has a copy. I use ChronoSync on the Macs, and tells it to "Synchronize Deletions". This means that the network disk ends up with a directory ("archive") which effectively contains the old files, the garbage.

However, with a centralized iTunes library (see post 003) it's different. You would have only one copy. So you would need to mirror or backup or synchronize your network disk with yet another disk! And once you find out how to, this turns out to be A Good Thing for all your files.

Try having another disk, of equal size, to the network disk stored at a friend or neighbour or at work. In another house. Then you would be safe against burglars and fires, while the disks are separated - so make the period they are in the same house short. However, with this scheme there would be lag. Synchronize 4 times a year, and you would loose max 3 months. A combined scheme with local synchronized copies then helps you almost to 100%.

I use ChronoSync also for the backup copy of the network disk. I "synchronize" all of it - quoted because I only do it for that disk from the main to the backup - one way. Having snapshots of the whole disk, times 'n' soon fills up your backup. With synchronization, the backup proper is increased or decreased in size along with the original system. Only the garbage directory grows - but you could delete that "archive" whenever you want to.

Observe that backup of the bootable parts (with system files) of your machines is an altogether different story. And backup of programs as well. However - most of that stuff has not been generated by you, so it's very probable that it would be possible to recover them. I mean, Apple iWork or Microsoft Office you would find on the original DVDs.

On Mac, Time Machine would help me with all of the factors above. When I installed the network disk, the Mac OS X version Leopard asked me if I wanted to use that disk for Time Machine storage. It's very nice, but I declined. Since we we also have earlier Mac OS X versions (Tiger) on the network, and since the hand-scheme I use serves us well, I decided to decline. And I don't really need to recover my system 143 weeks ago - and would want that storage space to be used for my daily more or less linear increase.

A psychological question now seems quite appropriate: wouldn't it be good for mental health to actually have a broken disk once or twice in a life time? Or even more often? Why were we supplied with the ability to forget? One thing is certain, some times that is necessary, too. And the controlled version of a broken disk: to throw away. That, too, is a privilege - some times.

4 - Airport and cabled mounting of disk is not seemless ::

If I connect my Airport Extreme network disk by cabled connection, then unplug the cable with the Airport active on my Mac OS X 10.4.11 Tiger iBook, I get a long period when the cabled connection will time out. This makes the file system non-responsive for that period.

I would like the Airport to take over this connection dynamically. I have a mental connection to the network disk. The operating system should make it like this for me - and to the client programs. (It could, immediately ask me if I wanted to switch to Airport (or back to cabled), if a "finger" is thought necessary in the handling.)

Filed to http://www.apple.com/feedback/airportextreme.html on 29 July 2009.

5 - Time Machine backup on Airport Extreme's USB connected disk ::

This chapter is obsoleted with this fact: "Apple does not support Time Machine backups to a drive connected to an AEX". See http://discussions.apple.com/thread.jspa?messageID=12975885#12975885 (started chapter 6 below, date 23Jan11). I also updated with this info on Wikipedia: http://en.wikipedia.org/wiki/AirPort#AirPort_Disk

6Jan11:
Mac OS X 10.6.5 Snow Leopard mac Mini
Airport Extreme serial number 6F8390GLYEW version 7.4.1 (tried both 7.4.2 and 7.5.2)
Using AirPort Tool 5.5.2 on Mac Mini
Decided no to use corresponding AirPort Tool 5.4.2 from Tiger in fear of compatibility problems
Network disk is Western Digital myBook 1TB type 0x11102 connect to USB on Airport Extreme
20Jan11: (see log in chapter 6)
Airport Extreme 7.5.2 (75200.14 when I press the version number)
22Jan11: (see log in chapter 6)
Airport Extreme 7.4.2 (74200.9 when I press the version number)
Airport Extreme 7.4.1 (74100.25 when I press the version number)

Had several problems with destroyed Time Machine image on the network disk. Lost with Airport Extreme 7.4.1 once but when I changed to 7.5.2 I lost again almost immediately, so I went back to 7.4.1.

It is very bad to loose the Time Machine backup! With this experience it's not a good idea to use Time machine as an archive, only backup! I use ChronoSync for archive!

By intuition I though that I could try yo move the disk cable! When I connected the disk to Mac Mini, Time Machine allowed me to start all over on that disk. I didn't. But when I put it back to Airport, Time Machine could not see it. If found a recipe on MacWorld: Take the disk back to Mac Mini and run Disk Tool and let it verify and reapair. I did - and the disk had error! The Extended attributes file had wrong number of access control lists (4949 instead of 4952), whatever this means. I let Disk Tool repair it and plugged it back into Airport. Now Time Machine saw it, exactly like MacWorld had said!

I am now running Time Machine anew with the 6Jan11 versions, crossing my fingers.
.
6 - Airport Extreme version juggling ::

This chapter may be related to problems described in intro above, chapter 5 above, and to 020 - Mac OS X network disk partially seen. I will try not to repeat, but update new things here. However, there is much version and config info up there.

Summary

OLD IS BEST: Airport Extreme old 7.4.1 seems more stable than 7.4.2. The newest 7.5.2 has crashed and restarted on some occasions ("much" traffic). Presently running 7.4.1
REPAIR DISK: Network disk needed repair (why?). May this have been the cause of Time Machine backup lost? See chapter 5 above.
MOUNTING: Mounting the network disk for iTunes on Snow Leopard must be done by "connect server", not double-clicking. See 020 referenced above.
DUBIOUS: There may be a problem with different AirPort Tool client versions if I used them at the same time on different machines. Now only using the newest 5.5.2, on Snow Leopard. How about two of same versions up?

MyDiskAcidTest is that I press "New Folder" on my network disk. If unkown error (-50) then it failed, elso ok with a new directory!

Log (newest bottom short log in (see log in chapter 5))

6Jan11. Chapter 5 above done.
7Jan11. The Mac Mini is always on, never idle. But when I came back to it tonight, the network disk was gone (for the millionth time), and I had to unplug/plug it the Airport Extreme 7.4.1. I had not used any Airport tool in between. Time Machine had a compliant up that it had not been able to run the backup. But the two network disk icons were there, I was able to unmount and mount again ok. But MyDiskAcidTest failed.
8Jan11. Same problem as yesterday: Time Machine had failed and MyDiskAcidTest failed. Unplugged/plugged the USB cable in Airport Extreme, then ok. I decided to look further at the MyBook Home (WD10000H1CS-000) network disk.So I downloaded from Western Digital's page here. It really said to update from Tiger (10.4.x) or Leopard (10.5.x) but did not mention Snow Leopard (10.6.x). I chose Tiger (even if Snow Leopard probably would have been ok). Then I went into Activity Manager and killed two processes named WD_*, I could see that they were both some Western Digital button manager etc. But I did not write it down. I then ran Disk Tool and verified, and there appeared an error again! I fixed as with chapter 5 above. The Tiger Disk Tool said it fixed it but then summed up in red that it was unfixable! Anyhow, I then updated the MyBook Home with WD_MB_Home_1034 (1.03).app that had to be run from the downloaded image. The program said update was successful. I then ran Disk Tool again and now all was fine! I did not "Download and install the latest WD Drive Manager program" as in the recipe, as the disk appeared fine and passed MyDiskAcidTest. Another observation is that the disk needed repair even if I had stayed at Airport Extreme 7.4.1 and Airport tool 5.5.2 on Mac Mini all the time!
19Jan11. Still working (since 8Jan11), after almost a week! Time Machine works, iTunes runs continuously, and the Airport Extreme also runs!
20Jan11 - Time Machine stopped! However iTunes ok, MyDiskAcidTest, wifi ok! Remember that last time (8Jan11) MyDiskAcidTest did not work! So now only Time Machine stopped! So, updating the disk firmware may have helped? But Time Machine could show the files, I could restore - but it complained that it could not go on with the present data base after having verified it. So, if I now had trusted Time Machine, my 28 GB would have been lost! Now I try Airport Extreme old 7.5.2 again.
3 hours after. Wifi died! Easy to see on all equipment. But MyDiskAcidTest and wired network was fine. In my bewilderness I opened the AirPort Tool, and it showed green status. So I pressed the green "light", and it reported "no problems". After this, wifi came back. The physical LED had been green all the time.
22Jan11. Airport Extreme 7.5.2 restarted several times, I think during copying to the extern disk. Another thing I had noted was that the disk never spinned down, which it had done with 7.4.1! AndYellow and blinking yellow LED. "Initialized (7.5.2)" or something like that in log. I lost Wifi and wired ethernet, and internet. It crashes so much that I'll try 7.4.2 (not 7.4.1 again now). Time Machine and MyDiskAcidTest are ok when the router is up. But what happens with the Time Machine data base on the network disk if it fails during updates?
22Jan11. 7.4.2 just crashed almost immediately! Giving up, reverting to 7.4.1.
CONCLUSION for now: going for 7.4.1, switching off Time Machine on the Mini, and will run it by hand. However, I will use ChronoSync for real backup work, since Time Machine with the network disk simply cannot be trusted. Also, will have to mount iTunes disk by hand, see Post 020. Will not throw away more time before Apple updates 7.5.2 to something that works. Updating the MyBook network disk firmware helped, see above - so it's not been a complete waste of time.
23Jan11 - yes, the network disk now does spin down again, within some minutes after last usage. 7.4.1 is best.
23Jan11 - reported this to http://www.apple.com/feedback/airportextreme.html with this text (minus links to here): "My conclusion is that Airport extreme 7.4.1 is the only present version that I can use. 7.4.2 or 7.5.2 will not work. Even with 7.4.1 I have to do Time Machine manually, so that I hopefully will not loose my backup. Even then I do ChronoSync backup to make sure, since Time machine cannot be trusted. iTunes with that network disk is now ok running continuously, as is general access from other machines - even if I have to mount iTunes' network disk by hand. And now the network disk spins down when it's not in use like it should. Airport extreme serial number 6F8390GLYEW. Before I updatde the firmware on the MyBook disk, also 7.4.1 was unstable."
I also posted the same as a support discussion, see http://discussions.apple.com/thread.jspa?threadID=2730728&stqc=true. Maybe there is some help to get 7.5.2 up again?
Have a look at this: "Apple does not support Time Machine backups to a drive connected to an AEX" (ref http://support.apple.com/kb/HT2421?viewlocale=en_US "Uses for the USB port of Time Capsule, AirPort Extreme, AirPort Express"). I replied in the post that "Another thing, if Apple does not support Time Machine on an Airport Extreme eXternal disk AEX - then why do they allow me to select it? And run the animal every hours for a week or three, let me extract files from it, until one day - bang!"
2Feb11 - Wifi stopped after 1.5 week! I had to restart the Airport Extreme (7.4.1, no use of Time Machine).
21June11 - trying 7.5.2 again. Just a temptation..
27June11 - reverting to 7.4.1 again. Same problems as described above. Any surrounding updates hadn't fixed the problems.
Filed to http://www.apple.com/feedback/airportextreme.html on 27 June 2011.

7 - Backup of the network disk was only a partial backup! (Access problems) ::

The network disk (figure, see Post 019) I synch by hand with ChronoSync (Mac), so I may loose the disk at any time without loosing anything within a week or two. Even the iTunes library I do a one-way copy into one of the permanent machines.

To complete the backup regime I use a disk the same size as the network, located in another house, which spends its life unpowered inside a plastic box meant for the freezer. Two to three times a year I bring it home and mirror the running disk onto it. In the daytime, while I'm home.

Both are 1TB disks, so I don't want to copy over the network. I disconnect the USB of the local disk and connect directly to the Mac Mini. The external drive I connect to FireWire on the same.

However, I only now solved a problem where, what's happened is that I really didn't get an exact copy. I had tried to copy with Finder, but some problem had it stop. And Finder stops after an error report, which ChronoSync doesn't. So I have used ChronoSynch with Backup Left-to-Right or Blind Backup Left-to-Right, but it complained that I didn't have access right to some of the files (reading access right, even if it looks from the log like I had writing access right problem).

This must be because the directories have been created from any of the connected machines, while it stood as a network disk. Since the disks now were connected directly, I tried to change the access right recursively at the top level for the disk as a whole. There even was an Ignore access rights to this disk, but none of them solved my problem.

What worked for me (disclaimer: in perspective - don't do this - see quote below) was to set the access rights to each of the few top level directories, recursively on subdirectories. It took a while. To start afresh I formatted the destination disk. Now, copying the some hundred GB of data went like a dream! All came over, no files were lost (I hope!). It seemed to work fine when I reconnected the network disk to the Airport Extreme again. iTunes worked fine, Time Machine etc. And ChronoSync from the other machines with the network disk.

Quote from an e-mail from somebodey who knows better than me:

- What I do know is that a 'network' drive connected to an Airport Extreme acts as Network-Attached-Storage (NAS) and is managed by the AFP server built into the Airport Extreme. All files and folders created on the NAS volume have ownership, group membership, and read/write/execute permissions set by the Airport Extreme and AFP.
- Connecting that NAS drive locally to a mac can deliver unexpected results. If I were to attempt such an operation I would most certainly not change all the ownership, group membership, and read/write/execute permission recursively on the volume.

By the way, we also some times send the best pictures to family, a pleasant backup type! And wo don't use Dropbox or any other file hosting services yet. And music is here, no streaming yet.

8 - iTunes 10.4.1 and Snow Leopard 10.4.8 with destroyed files ::

(Oct 2011) -8084, -8072 etc. unknown error codes

I tried to iTunes-synch the the iPad with the picture library I now had had for years, on the network disk. It complained that it could not synch the pictures, that I did not have correct access rights for the some 450 last pictures. I tried again. Same problem. The iPad picture library was acomplete mess! Pictures in wrong folders etc. Then I decided to move the source library into the Mac Mini and observe how it failed.

One jpg file seemed to have an IPCT digest code that Mac OS or Finder or iTunes did not like(?). When I copied the group of files individually I got them across!

Then, some of the jpg files were unix executables from Snow Leopard, and not accepted by iTunes. But Tiger saw them as proper pictures and I saved them under new names, deleted the old (not allowed from Snow Leopard), and named back to original.

Finally I was able to get a clean 2804-numbered picture library that iTunes again synched!

I seem to experience increased problems with the Airport Extreme controlled MyBook disk. I don't know what to do.

How about a Time Capsule with built-in server grade disk? Will it come with Thunderbird? Hmm.

9. Moving from Airport Extreme to Time Capsule ::

See http://oyvteig.blogspot.com/2010/08/019-play-with-network-disk-based-itunes.html#fig2 for the figure.

There were several reasons for the move:

A while now I have been running iTunes on the Intel Mac Mini, and its Time Machine backup software will not work correctly on the disk that I had connected to Airport Extreme [1]. Time Capsule is made for it. Apple "forced" me to buy an Apple product. As a programmer I know how complicated making things work across platforms is, but then I may be too forgivable.
The Western Digital myBook external hard disk made too much noise when it at least once per minute seemed to be accessed. It didn't seem to spin down ever, especially not since I started to have iTunes always run (I didn't play all the time!). This may not be a blame on myBook, but I did't like it. Time Capsule spins down under the same circumstances. And I can't really hear it when it runs.
I had updated the firmware on myBook, but in a way there was some problems left. The result seemed to be the "Airport Extreme version juggling" above. Now I run Time Capsule's latest firmware version. It seems to be ok.
I had problems with access to myBook (read: Airport Extreme'sn USB connected hard disk) from iTunes (on Mac Mini with newest OS) and iBook running Tiger. Picture directories for iTunes synch had to be created from the Mini, but could be filled from Tiger. I also saw some files turning up as "unix binary" as seen from the Mini, but perfectly ok jpg from Tiger. I hope this will not be so with Time Capsule.
Lots of pictures have turned up to have been created in 2013, some two years from now.
Synching with ChronoSynch between the machines started to take time. I could see 8 minutes. Not it's under 2

I studied Time Capsule's supposed heat problems, but decided to try it. The place I mounted it, it has never become as hot as the Extreme, even if max dissipation is 34 Watt for the Capsule, some 13 Watt more then the Extreme. I mounted Capsule below all units, and it has freen air underneath, and is held by a solid aluminum bar, that I thought would help cooling. I have never hear it start its fan, which I think is there also at this version [2].

I think our internet connection has become faster. It feels like it, even if the service provider has not upgraded.

[1] - See http://en.wikipedia.org/wiki/Time_Capsule_(Apple)#Apple_AirPort_Extreme or http://support.apple.com/kb/HT2421?viewlocale=en_US

[2] - http://www.ifixit.com/Guide/Replacing-Apple-Time-Capsule-HDD/2750/1

10. How I moved iTunes from the one external drive to Time Capsule ::

This chapter is not a recipe. Apple has the recipe [1].

But you might encounter problems. I did. I also did at least one thing wrong.

The new iTunes directory structure [2] is very clean, as opposed to the earlier. I wish I had looked at [2] and [1] again before this exercise. But when the iTunes config dialogue asked me to give the position of the new "iTunes Media" directory, I thought that since I had always there pointed to the iTunes base directory, that Apple now had decided that it should be called "iTunes Media". So I made the new position on the Time Capsule disk to be "iTunes Media". When iTunes had moved the files another "iTunes Media" directory had appeared underneath. It was empty. It started to smell.

Lesson one: the root iTunes directory is still "iTunes". When you want to tell iTunes of the new position, make this "iTunes" directory - and a directory called "iTunes Media" under it. Then, point iTunes to the latter! Observe that an "iTunes Music" directory is gone, it's now "Music" under "iTunes Media".

This is a change in the iTunes config dialogue box semantics that Apple did not get across to me! It would have been easy to either explain, or just let me point to "iTunes" and they could have added "iTunes Media" themselves.

This seemed to cause a lot of problems. But I had a backup copy of my complete iTunes system, which was nice. But before that, iTunes all of a sudden told me that my "iTunes Media" file complete path had a "-1" or "-2" in the part of the path that contained the name of the Time Capsule's internal disk name. Like, instead of "CAPSULEDISK" it would be "CAPSULEDISK-1" or "CAPSULEDISK-2". This was complete nonsense, and iTunes then could not find the place where it should find the files.

I did not fix this before I had started all over, with correct "iTunes Media" pointer, and the old data iTunes base files. Now iTunes moved over and placed all "media files" in subdirectories in "iTunes Media". Fine! After this I stopped iTunes. Moved data base files to just under "iTunes". Then started iTunes by holding down the command button. Fine! (According to [1] it seems like iTunes do move these files?)

Almost fine! But some of the files were not seen! I could not understand a thing, so I exported the library to a text file. I inspected it and searched for the old file position, like "MYBOOKDISK" for13 files that iTunes had told me were not there, when I tried to access them. I noticed that I could not open the directories they were in, because I didn't have access. The myBook disk was now not connected to the parked Airport Extreme, but to the Mac Mini. I changed access rights to them, and moved them to the new Time Capsule disk. I then (after lots of fiddling), I deleted them in iTunes, and imported them anew. Voila!

But iTunes hadn't said a thing! If I hadn't been so accurate and visited every song (open list mode and arrow down and don't expect and exclamation mark), my directory could have rot on root. Apple has chosen to inform when a song is accessed (playing or synching). But, I wonder what the reason is for not telling during moving?

Another thing, keep the "sentinel" file [3].

Reported to http://www.apple.com/feedback/itunesapp.html

[1] - iTunes for Mac: Moving your iTunes Media folder (Windows)

[2] - What is the default iTune Folder Structure for iTunes 10? (see graphics somewhat down)

[3] - What is the Sentinel File in the iTunes Directory?

Afterword

When I wanted to synch the pictures, iTunes failed for some reason. The pictures again should be on the external drive. I deleted the "iPod Photo Cache" directory and restarted. No help. After two hours of trial and not succed, I tried to move the last synched picture directory from the Mac Mini to the external drive. I also deleted "iPod Photo Cache" there. Only then did it complete the synch, with pictures on the iPad again. Please, Apple - say something understandable in a dialogue box when things fail!

I did get an "unknown error 1630" that iTunes "could not search for updates for operator configuration for iPad" (translated from "iTunes kunne ikke søke etter oppdateringer for operatørinnstillingene på iPad. Det oppsto en ukjent feil (1630)" in Norwegian.) I don't know how this should have influenced picture synching.

Analysing afterwards. I had made a copy of the whole Time Capsule picture directory, and now wanted to delete it. But there was one file that could not be deleted, since it was "used", Lion said. This was not so, it was a jpg having become a "unix binary" (see chapter 8 above). I saw that this was also present in the directory that iTunes had used to synch the pictures, but that "picture" was not synched, but the rest was ok. When I looked at the file from Tiger - this time it was also a binary. But as I did not succeed in deleting the file from Lion, I was from Tiger. I have no idea if this was the problem, but since iTunes did complete the synch, my guess is that this was not the problem.
.
.

003 - Shared iTunes library on a network disk

2009-01-22T18:35:00.090+01:00

Goal

This post is for you who want to move all your music, video, AppStore programs etc. - or rather, your iTunes library - to a network disk.

Inconcistent iTunes versions will probably kill this architecture! (Sept. 2010)
I now have left this architecture (Oct. 2010, see (*) below)

With iTunes 10 running on one machine and iTunes 9 on the two other machines (iTunes 9.2.1 is still "newest version" for the G4 PPC Tiger machines), the iTunes library files on the network disk is bound to be in conflict between generations of iTunes (soner or later, I'll update when 10.0.x). I will certainly choose the newest, and I will then instead run the two older iTunes' sharing the music from the one newest. Apple use their Digital Audio Access Protocol (DAAP) for sharing, it won't be incompatible across versions. I would then for my usage let the Mini (see post 019) run all the time.

I would certainly have to learn if I would have to sync the iPhones, buy music etc. on the newest iTunes - if I only want the newest to "share out" its library. The usage would probably have to change. But the good thing is I would then be inside Apple's intended usage!

(Sept.'10)

(*) - One iTunes shares out to the others

When Apple parked the iTunes version on Tiger to 9.x.y with respect to the flow of newer versions, in Sept. 2010, I was forced to leave the architecture described in this note.

In retrospect I don't understand why I bothered doing it the hard way, when Apple in fact have a better solution. There are limitations to Apple's architecture as well. Like start-up time and the lack of CD covers on the slaves.

I think that I had been sceptical to two things: having the Mac Mini run all the time and the network disk (I did not drop it!) having to spin all the time:

However, after I started Jack OS X Server (See "019 - Play (with) network-disk based iTunes library?") to run continuously on it, I saw that it was no problem to run the Mac Mini all the time: there is no fan that I can hear at least.
Also, running iTunes all the time would still let the network disk enter idle and stop when I didn't play music (and at the sime time there was no other activity)
Now, have a look at Post 019 - it's getting better, now also using Apple's remote App! All of a sudden it's easier than old CD's.

A plus with the Apple's way to do this is also that I may now have copies of some of the music on the slave machines.

When going to the Master / Slave architecture (and not pseudo multiuser client), I had to move the iPhone synch directories to the newest machine and newest iTunes. This was some job, but I "only" needed to understand what files are where and move accordingly. I am not certain if iTunes has a function like "Move iPhone synch to another machine". This might be very nice to have! (Reported to http://www.apple.com/feedback/itunesapp.html)

A disclaimer and four warnings

Disclaimer: I take no responsibility for any errors in this post. Your iTunes library is valuable, so you should read this text with caution. A good idea is to backup your iTunes library before you make any move.

The first warning is that if you move your iTunes library from, say, your laptop - then you have moved it! In other words: if you bring your laptop on a vacation - your music stays home! However, if you have synchronized your music with your iPod or iPhone - you still have the music with you. (You may keep a copy of the iTunes library on your machine - if you want to make things worse for yourself as the months pass.)

The second warning starts with one good news. On the vacation you may upload iPhone pictures to your laptop (substitutes for not having your proper digital camera available). Uploading is a mechanism outside iTunes (Image Capture on a Mac). However, if you want to do more with iTunes and your iPod/iPhone on that vacation - go ahead. Just follow iTunes' instructions when it cannot find your iTunes library. But - what do you do with the two libraries you have when you come home? The one on the network disk and the one you, on that vacation, have brought up from nothing on your laptop? Do as suggested here? I have not tried it, but I am not certain if I want to.

The third warning was about Front Row. See below.

The fourth warning. Read Switching between iTunes machines (below) for a major drawback!

You should now be warned, and may read on. I have moved our library, and have not regret it.

The machine park

Our machines are three, all Apple machines, two with Power PC G4 processor running Mac OS X version Tiger and one with Intel processor running Mac OS X version Snow Leopard. All run iTunes 9, latest build. This post is in that context. There is a figure my note 019.

The problem description below is from iTunes 8 (2008-2009), but I have no reason to believe that newer versions, like 9.0.2 that I installed recently has helped. There also is a new effect with 9.1 (April 2010, see below). iTunes 10 will also soon be tested. Apple notably have fixed some "sharing" problems, but this post is not about cross-iTunes "horizontal" sharing, but about to-network-disk "hub" sharing.

Moving iTunes files: The Reference

I moved the files according to this recipe: www.ilounge.com/... (Transferring your iTunes Library) written by Jesse David Hollington. You should read that document first (but see next paragraph even before that), and perhaps move files from one machine only. Then you could come back and read the issues I discovered when I moved iTunes libraries from three separate machines.

If you just need the how, and not the why, go to Apple's excellent pages "iTunes for Mac: Moving your iTunes Music folder" or "iTunes for Windows: Moving your iTunes Music folder".

Another ok reference is this wikipedia article about library sharing.

The first move

The first move of files I did following the above chapter. It goes something like this: 1.) First set the iTunes Music folder location and 2.) do Consolidate library and let iTunes work for a while, then 3.) stop iTunes and then 4.) move the iTunes database files (not the iTunes Music folder, it has been moved for you in point two) to the network disk, then 5.) press the OPT key and start iTunes (keep it depressed until you see iTunes) and 6.) tell iTunes where the new iTunes library now is, namely on the network disk.

After that music plays perfectly - from the network disk - on all machines, but one at a time (see below)!

The second and third moves

On these machines I saw that I did not have to move the iTunes database files, as iTunes did that for me, and it merged the new contents into the network disk's database files. This had to be like this, since for each database type of file, the now central iTunes library would only have one of each. For the first machine I had done this manually.

Multi-user access: one iTunes at a time

As Hollington points out, iTunes will not treat multi-user access according to the book. It simply is not built that way by Apple. Not yet. I think this means that even playing some song on two machines at the same time may fail, since iTunes writes to some database file how many times a song has been played. It may be a good idea to backup the database files from time to time. Multi-user access would allow only one user to open the files for writing.

Tiger vs. Leopard on multi-access

What I discovered was that while Tiger machines would happily open as faulty multi-user, Leopard halted me and said it could not open the database because it did not have the correct access rights. True! I then quit the Tiger iTunes and then started the Leopard iTunes - normally. I wonder why Apply did not do this for Tiger.

However - even if I, in Tiger, am allowed to start iTunes and play songs when another iTunes also plays, iTunes still seems to protect the database files from faulty writing. When Tiger iTunes tries to write to one of them (to increment the play count?), then Tiger iTunes will tell that it failed to write to the file. This is really good news. So, is it less error prone than I thought?

Reappearing iTunes files

On one of the Tiger machines I saw the iTunes database files reappear on one machine. However, it played from the network disk. I then did just like I had done after I had moved the files: i just renamed the iTunes directory to iTunes_somename to make iTunes not find it. Then, when I knew things worked, I deleted the local directories.

AppStore applications and their data lost

On one of my synchronizations after this, where the iPhone (2.2.1) was synched with Tiger iTunes (8.0.2) - some of my iPhone applications got lost! One of them was Tuner Internet Radio from Nullriver Inc. I had observed that iTunes had shown me a list of apps to synchronize, but had really not paid attention to it. So, iTunes did what I told it - synchronized - and then removed some that were not in the list! No warning that that list was smaller than the (card layout) list in the Programs (Programmer in Norwegian). And I had not noticed.

In my despair I double-clicked the icons in the Programs window, and then iTunes told me it could not find the original app executable file. I handled each of the programs, only some of them were lost. Then I pointed iTunes to the external network disk, where the apps had stayed since my change to using a network disk, one app after the other. Having finished all that, I went to the synch window again, to learn that iTunes now wanted to synchronize the full number of apps. Good, so I synched.

Only to learn that my list of Norwegian radio stations was gone!

iTunes tip: for those with a shared network drive - count the number of apps you have, and don't let iTunes synchronize any one of them away!

I have no idea why iTunes had lost track of some of my AppStore apps' original files. Apple, do you hear me?

Backup of the network disk

This is quite important. Read on in my post 004 - Living with a network disk in the house.

Front Row and network disk: is a match!

- Latest update (29July2010): iTunes 9.2.1 and FrontRow 1.1 (all the time here the context is iTunes everything on the network disk) - FrontRow at first refused several times, showed the CD covers, but could not play the music. After a while it did play the music. But all of a sudden I got a crash message from iTunes, which I had stopped in order to get FrontRow up. Network iTunes is still not arrived completely
- Older update (8May2010): FrontRow 1.1 and iTunes 9.1.1 running in Mac OS 10.6.3 Snow Leopard now works perfectly! I don't know exactly when the change happened, or which componennt that did the trick, but it works! Airport Extreme is still 7.4.1 (not the newer 7.4.2). (Even this reported to http://www.apple.com/feedback/itunesapp.html)

- Even older update: this was fixed in Front Row version 2.1.7 (11Mar09) - but it was in fault again half a year later running Leopard. It also does not work on Snow Leopard 10.6.2, Front Row 9.0.2 - where iTunes 9.0.2 certainly works perfectly well. I still have Airport Extreme 7.4.1, as 7.4.2 had to be removed due to intermittent failures. (Reported to http://www.apple.com/feedback/itunesapp.html)

A month into the move I discovered that Front Row does not play my music any more! This is status with all latest updates Feb. 2009. A month indicates that I don't use Front Row very much, but I certainly was planning to.

Where Front Row could share any iTunes library from any other machine seamlessly, it now just tells me that it can't find any music, and that I should import or buy music.

According to http://en.wikipedia.org/wiki/Frontrow, Front Row does not support the necessary protocols for network media. So, it must be a matter of time. (So, with the update above, it's been fixed!)

Version conflicts

When I updated to iTunes 8.1 (March '09) on two machines, I had to update on the third. The old iTunes politely told me that the library was from a newer version, gave me no option - and quit. This is as it has to be - making file formats forward compatible is impossible if Apple, in this case, has decided that the user (I , in this case) must have the new functions. Even on lessser criteria it's hard to make file formats forward compatible.

September 2010: iTunes Library conflict between versions 10 and 9? (See "Inconcistent iTunes versions will probably kill this architecture!" (above) and note 019 for a figure and further updates about this) (Also See Apple's iTunes download page which includes G4 hardware but excludes Tiger 10.4 sw, my question and Phil's)

iTunes tip: when you download the new version, place the installable file (.dmg for Mac OS) on the network disk - and use it when installing on all machines. No need to download it many times!

What is an "iTunes server", some times installed on third party network stations?

I queried this to a discussion group at ReadyNAS (www.readynas.com/forum... - You must get an account to read the thread). I had read about "iTunes server", even an "iTunes client" for Netgear ReadyNAS Duo and HP MediaSmart Server. A person there helped me fill in the gaps. Thanks, stretch! I will try to summarize:

First, it's not for me: With my use of iTunes, iTunes and nothing else (like Java Firefly client), I would loose more than I would gain in running the iTunes server. The units may be well worth a buy, but I think, not for the iTunes server. The only thing I, in my opinion would gain, is the possibility to play the music files from more than one iTunes, simultaneously. But the downside is limited functionality and a lot of manual handling of files. (See above when iTunes also may behave like this)

An iTunes server is most often a free Firefly Media Server (http://en.wikipedia.org/wiki/Firefly_Media_Server ). It understands the protocol that iTunes instances use to talk with each other: Digital Audio Access Protocol (DAAP) (http://en.wikipedia.org/wiki/Digital_Audio_Access_Protocol)
A unit that runs Firefly will appear as a shared view of the music library, as seen from inside iTunes. This has been mentioned in Comment 3 (below). It would look like any other shared iTunes library - with limited contens.
Only music. Provided you have not bought it on iTunes Music Store (iTMS).
If you use Firefly you must keep at least one real iTunes up and running to use when you purchase from iTMS, import (why do they call this "rip"?) CDs, create playlists, synchronize iPhone, iPod etc. That machine will be your management computer. And Firefly will never hear a word from iTunes about this.
After having done the above, the contents would appear on your management computer. Remember that with the solution I have outlined above, all contents would appear on the network disk, and appear on any iTunes machines of mine. This is the main rationale for me to conclude that Firefly is not for me.
Should you later decide to move the contents, you must do this by hand, provided the same as above. However, not the mpg (imported and -converted inside iTunes) movies you took of your grandchildren. Follow some of the procedure outlined in this post.
Shold I have gone for Firefly, I would have ended up having something here, there and everywhere, including my shared network disk. It may be ok for some. However, I would wait for Apple to put a real iTunes server on my Airport extreme. Apple?

Switching between iTunes machines

iTunes 9.1.1 "gapless playback" update triggered at swap between Tiger 10.4.11 and Snow Leopard 10.6.3. No fix for iTunes 9.2 on 10.4.11 and 10.6.4

iTunes now too often goes to Apple's server to query about all our thousands of songs' non-stop playing information. This takes minutes! (iTunes 9.1.1 (11), April 2010 - on all machines).

First iTunes updates its data base library:

During this time I'll have to wait. Much longer than I appreciate. iTunes does this every time I switch machine, and it won't play any music while I wait. On a Lamp Mac with latest Tiger it now uses 10-15 minutes for this update! On the iBook with latest Tiger, it's about 5 min. And on an Intel Mac Mini witha latest Snow Leopard it's a little less. These figures represent a major obstactle to using iTunes with a network disk music library. It's bad!

Then goes on with this "getting information about non stop playing" (*1):

.

During this much longer period I am able to play, but it still is annoying. This second phase is new to recent versions of iTunes (April 2010).

I think this is done after iTunes updates its data base. But I am sure that it is triggered when I swap between Tiger and Snow Leopard playing. Observe that our iTunes library is on a shared network disk, and only one instance of iTunes will ever run.

Update: If I try to stop iTunes while this is running it tells me "processing album pictures" (*2). This perhaps gives a hint of what's going on.

(*1) The Norwegian text is "Finner informasjon om oppholdsfri avspilling" - "Getting information about non-stop playing". A comment (below, thanks Hallvard!) informs me that this is "Gapless Playback" in English.

(*2) The Norwegian text is "iTunes behandler albumbilder. Er du sikker på at du vil avslutte iTunes nå?"

Filed to http://www.apple.com/feedback/iphone.html

History

I will keep you posted for any interesting things I may experience, or any news that may cross my way. However, the reference above would probably outdate this post fast! So, go there, as I could not and will not mirror it. Updated 25Sept2010.

002 - Hjemmeruter og CanalDigital (in Norwegian)

2009-01-14T20:11:00.029+01:00

Canal Digital og hjemmenett ("bredbånd")

Dette er et lite notat om hvordan du kan konfigurere ruteren din dersom du får internett fra Canal Digital over kabel.

Denne bloggen gjelder både for Thomson modem TCM420 (produsert i 2008) og TCM470 (produsert i 2010). Det siste fikk vi tilsendt etter at den første takket for seg etter nesten tre år, i mai 2011. Vi mistet nettet, og enheten lot seg ikke resette. Canal Digitals support var upåklagelig, men det tok for lang tid i posten - en dag mer enn de sa i support. (På lørdag kveld da jeg ringe dem sa de mandag kveld, men det kom tirsdag kveld, levert på døra av posten.no.)

Denne bloggen er fra 2009, men jeg har oppdatert den i 2011.

Canal Digital og tildeling av ip-adresser

Canal Digital sin hovedserver tildeler ip-adresser ute hos kundene, maksimum fire adresser. Thomson-modemet som de sender til deg i posten utfører ingen tildeling av ip-adresser. Dersom du har en ruter, en printer og to maskiner, har du alt brukt opp fire.

Det er ikke lurt å la systemet stå slik. Det åpner for konsistensproblematikk i det du skrur enhetene dine av og på. Det vil ofte se ut som om du har mistet nettet!

Apple AirPort Extreme

Hjemme hos oss benytte vi AirPort Extreme som ruter (se Wikipedia). (Den inneholder også en disk-server knyttet mot en USB-utgang. Dermed kan man koble opp en sentral nettverksdisk (eller fler: bruk en USB hub). Så har den ingen synlige antenner, og trådløs-effekten kan settes helt ned til 10% for å redusere stråling. Den har dessuten ingen vifte, ikke verst for en filserver.)

Så var det oppsettet: "Del en offentlig IP-adresse" medfører at Canal Digital bare trenger å tilordne (vha. DHCP, ikke manuelt) en eneste adresse: den til ruteren. Så må du fortelle ruteren at den skal kjøre DHCP på hjemmenettet for å tilordne hjemmenettet sine adresser. Da må du gi den start- og -sluttadresser. Bruk et av forslagene som AirPort verktøyet kommer opp med. Deretter må du aktivere NAT porttilordningsprotokollen ved å krysse av på neste fane. Dette medfører til sammen at alle enhetene i hjemmenettet ditt bare benytter en offentlig ip-adresse mot Canal Digital.

Ferdig!

Husk å sette på passord, og trykk "Oppdater" dvs. lagre. Vips: etter ny oppstart vil du ha et nett som går som en drøm! Alle ip-adressene på hjemmenettverket ditt vil ligge fra topp-nummeret og ned, mens AirPort'en sin ip-adresse mot hjemmenettet vil være et nummer under startadressen. AirPort'en/hjemmenettets adresse mot Canal Digital er jo tildelt av Canal Digital, så den kan være hva som helst.

Oppskriften ovenfor fungerer også på AirPort Express og Apple Time Capsule. (Okt. 2011: Jeg har forresten byttet ut Airport Extreme med utvendig myBook WD NAS disk til Time Capsule med innebygget NAS disk. Se egen blogg om dette her. Du kan nok benytte oppskriften, også på din ruter?, uansett type.

Ferdig, eller?

Vær klar over følgende: hvordan NAT-adresseoversettingen i ruteren din er implementert er avgjørende for hvor godt dette funker. AirPort Extreme ser ut til å gjøre dette bra. Les om fordeler og ulemper ved NAT på wikipedia. En ting er i alle fall sikkert: dersom NAT ikke er skrudd på, funker og funker nettet ikke, du resetter og skur av og på og drar i kabler og står i, men ingen ting blir bedre. Sett NAT på, så går alt så mye bedre!

Canal Digital falt ut - og jeg fikk dette!

Dette er en feilsituasjon som tydeligvis ikke oppstår når Thomsom TCM470 modemet har nettverksforbindelsen i orden.

Svitsjer modemet til NAT når det mister forbindelsen med Canal Digital sin server? Eller, har modemet alltid NAT påskrudd, men det blir ingen adressekollisjon siden modemet bare har en forbindelse til basestasjonen? Denne nettverksforbindelsen behandles jo forskjellig fra forbindelsene til hjemmenettet.

Jeg gjorde ikke som det står, men ventet til Canal Digital fikk rettet feilen. Da forsvant begge feilmeldingene fra Time Capsule basestasjonen også. (5. nov. 2011)

Jeg har slått sammen to dialogbokser:

Ingen DNS-tjenere

Denne Apple Wi-Fi-basestasjonen har ingen DNS-tjeneradresser og kan ha problemer med å koble til Internett.

Dobbel NAT

Denne Apple Wi-Fi-basestasjonen har en privat IP-adresse for Ethernet WAN-porten. Basestasjonen er koblet til en enhet eller et nettverk som bruker NAT (Network Address Translation) for å levere private IP-adresser. Du bør endre Apple Wi-Fi-basestasjonen fra å bruke DHCP og NAT til bromodus.

Jeg ringte og snakket med support hos Canal Digital og traff en som hadde peiling. Han sa at for alt det han visste, så kjørte ikke deres boks NAT.
.